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From the Editor 
Tara S. Behrend 


I’ve been thinking about change lately, prompted by an exciting email | received in January. Shelly 
Zedeck was wondering if | had any interest in his collection of old TIP issues. At first, | hesitated. | was 
only in California temporarily, so a logistics challenge presented itself. How would | transport these 
treasures from the West Coast to the East Coast? How many could | fit in my carry-on before resorting 
to shoving them in pockets or under my hat? After only a few minutes of deliberation, though, | decided 
the airline acrobatics were worth it. | couldn’t pass up the chance to get my hands on this archive of 
SIOP history. In case you are wondering, by the way, I can fit exactly 12 issues of TIP into my laptop 
case. Only three can fit under a hat. 


In reading through the archive, the first thing | learned is that SIOP history is heavy. And | mean 
physically heavy. | assume that all the collected wisdom adds pounds. 


The second thing | learned is how very much TIP has changed since Issue 1. Each editor has a put a 
personal touch on the publication, most immediately evident in their choice of cover design (see below). 
| confess that | was jealous that T/P no longer has a physical cover, and thus | have no outlet to inflict my 
terrible taste on all of you. 


THE 
INDUSTRIAL PSYCHOLOGIST 


DIVISION 14 


ae 


APA 


NEWSLETTER 


DIVISION OF 
TRIAL AND ORGANIZATIONAL PSYCHOLOGY 


In these pages, | found deliberations about whether to change the name of the Society; reports of 
changes in US laws that affect our practice; celebrations of our best scientific minds; and throughout, a 
culture of respect and collaboration despite clear differences in opinions. Some things never change. 


This issue of TIP is one I’m very excited about, and one | hope future readers will refer back to with 
interest. In the summer of 2016 when I took on this editor role, one of my goals was to launch an open, 
preregistered, and transparent project to generate rankings of l-O psychology graduate programs. 
Whether you love rankings or hate them, people use them, and so it’s worth doing them in the best way 
possible. To me, that meant identifying measures of quality that previous rankings projects may have 
overlooked. A team, led by Nick Salter, put out a call for proposals, and five teams submitted proposed 
methodologies. A committee of experts reviewed the proposals, and data collection began in early 2017. 
| am delighted that the final reports from those teams are now available. You can find them in a special 
section of this issue. Each team used different indicators so it’s not surprising that they came to different 
conclusions about the final rankings. The truth about graduate education is that different programs do 
different things well. 


| want to thank the entire team who agreed to work on this project: Allie Gabriel, Joe Allen, Dave 
Sowinski, Loren Naidoo, and especially Nick Salter, who has led the entire project with enthusiasm and 
great insight. | encourage you, as readers, to reflect on whether you agree with these rating criteria or 
whether you could generate a different list. It is useful to remember that rankings of any kind make 
assumptions about what kind of work is valuable. Rankings also involve many “researcher degrees of 
freedom” —decisions by the rankers that may not be explicit in the final product. For example, if a 
faculty member switches programs, when should her publications be counted for program A versus 
Program B? No ranking list will be perfect, but this project gives us some different ways of thinking 
about excellence. 


This issue also has lots of other terrific resources and information for being your best self as an I-O 
psychologist. For example: In their final installment of Lost in Translation, Andrew Collmus and Mike 
Litano offer an in-depth and extremely valuable look at how to communicate visually with data. I’m sad 
the series is ending (for now), but I’m very proud to be able to publish their terrific work, and I’m sure it 
will have long-lasting utility for the field. 


This issue also bids farewell to the special series on the intersection between I-O and the field of 
Learning & Development, expertly led by Amy DuVernet and Tom Whelan. | encourage you all to spend 
some time with their column and think about what we can all learn about learning. 


April marks a time of change for the SIOP Executive Board and committee chairs, too, which means you 
will see other new faces in upcoming TIP issues as the reins are passed. Please thank these folks when 
you see them at the conference—it is a lot of work, done for very little reward, just out of genuine care 
for the field of l-O psychology. 


President’s Message 
Fred Oswald 


This fourth and final President’s Message is dedicated to two major issues that SIOP and the profession 
of l-O psychology squarely face today and in our future. Let me elaborate on them here and hope they 
stimulate our continued thinking, discussion, and strategic actions as a society. 


The first issue pertains to very notable and noticeable changes in our profession and membership within 
SIOP. Although these changes have been documented in the recent past (e.g., see Rob Silzer & Chad 
Parson, T/P, July 2015), they need to be reassessed more broadly and more often. To focus on one im- 
portant example that is already impacting our future: Current trends reveal rapid growth in the number 
of terminal master’s l-O programs, along with the high volume of I-O psychologists coming out of them 
and working in an increasingly wider range of professional settings. Concurrent with this trend, or per- 
haps because of it, many PhD I-O programs are beginning to incorporate terminal master’s programs, 
both on-campus and online. Ideally, terminal master’s programs can provide important support for I-O 
psychology PhD programs of the future—and not simply because deans like money (although this can- 
not be denied). Terminal master’s programs can serve as a stepping stone for undergraduates who at- 
tend the I-O PhD-granting university; they can connect practice to science more closely within PhD pro- 
grams; they can increase the community and visibility of |-O psychology within the university; they can 
identify students who might transition a PhD program; and they can create and expand the ecosystem 
of local applied and research opportunities. However, this does not happen magically; it requires very 
strong institutional commitment to appropriate resources and incentives for their sustainability. Many 
PhD-granting institutions have resisted implementing terminal master’s I-O programs for lack of these 
two major requirements. 


All of this growth in I-O psychology training sounds good, on the face of it. At the very least, the growth 
is consistent with US labor-market data indicating the value and demand for l-O psychology in the work- 
force (e.g., see https://www.bls.gov/oes/current/oes193032.htm#top). In this context, our I-O psychol- 
ogy education and training efforts have become more essential than ever—and fortunately, you might 
remember that SIOP has recently updated its Education and Training (E&T) guidelines that inform the 
nature and quality of our I-O educational programs. If you haven’t done so yet, check out SIOP’s 2016 
competency-driven guidelines for |-O master’s and doctoral programs at http://www.siop.org/ETguide- 
lines.aspx (with thanks to Whitney Morgan, Joe Allen, and the Education and Training Committee). 


These SIOP E&T guidelines are intentionally flexible. All |-O psychology programs tend to cover “core” 
competencies (broad psychological training, statistical and research methods, ethical issues); however, 
greater flexibility is accommodated when covering substantive I-O topic areas (e.g., organizational devel- 
opment, groups and teams, performance appraisal/management, training). I-O programs thus reflect 
natural variation that capitalizes on faculty expertise at local institutions. They are quasi-experimental 
settings that could be examined more closely. Although SIOP already gathers information on I-O pro- 
grams, more detailed tracking of the number, nature, quality, and timing of topics covered by I-O PhD 
and master’s programs will allow our guidelines and programs to inform one another better over time. 


SIOP is also tracking developments in I-O related fields (OB, LIR, organizational science), and it is clear 
we are improving in this direction as well. The newly established Future Scanning Ad-Hoc Committee 
(FSC; Alexis Fink, chair) has set up a cross-functional structure that connects to the External Affairs Of- 
ficer (Janet Barnes-Farrell, Chair; Madhura Chakrabarti, FSC liaison), Membership Services Officer (Mo 


Wang, Chair; Evan Sinar, FSC liaison), Instructional and Educational Officer (Milt Hakel, Chair; Kurt 
Kraiger, FSC liaison), and related SIOP committees to ensure that our radar is highly attuned to the edu- 
cational and professional landscape relevant to I-O psychology. 


The second issue deals with how we can better frame and communicate our activities as l-O psychologists 
to a range of others (e.g., managers, university administrators, granting agency program officers, high 
school students). This issue arose most recently in a lunch | had with Incoming SIOP President Talya Bauer 
and Joel Quintela (Quintela Group) to discuss the nature and future of our profession. We didn’t solve the 
world’s problems over lunch! Instead, Joel, Talya and | focused on one topic: the problem of how I-O psy- 
chologists, and I-O psychology, can be better recognized by organizational decision makers. We discussed 
how Team SIOP (my vision this year) was about framing our work in terms of problems and impact (Steve 
Kozlowski’s presidential vision) so that people from all walks of life can understand how I-O psychology 
benefits the future of the workplace (Mort McPhail’s presidential vision). Joel emphasized the need for not 
using $20 GRE words in communicating that message. (How perspicacious of him!) 


Talya and | relayed to Joel how SIOP is becoming much more problem focused and issue focused in its 
current advocacy and communications efforts to promote I-O psychology. SIOP understands why it is 
critical that we continue in this direction; for instance, we recently held a meeting involving SIOP mem- 
bers concerned with veterans affairs issues (Nathan Ainspan chairing a subcommittee within GREAT; 
coordinating with Jesse Poon and Bill Ruch at Lewis-Burke). This is one example out of many, where SIOP 
is becoming increasingly engaged in more problem- and issue-focused communication strategies. My 
presidential involvement has been in providing broad-based input and support to relevant SIOP Commit- 
tees and the Administrative Office in communications-related tasks such as (a) revamping the content 
and format of the siop.org website (stay tuned); (b) reconsidering the content, format, timing, and audi- 
ences for SIOP messaging across multiple media; (c) advancing our science and practice advocacy efforts 
with Lewis-Burke; and (d) extending our white papers into other informational documents and materials 
to serve a variety of audiences. 


Hopefully in this reinvigorated process of translating our work and connecting to others, SIOP will pro- 
vide our constituencies an interesting, useful, and accessible entry into our “science for a smarter work- 
place”—and maybe we'll only spend $40 worth of $20 GRE words! Three notable translation efforts of 
this nature (and there are others) are already underway. First is the new Oxford Translational Science 
Series edited by former SIOP president and current Research and Science Officer, Steve Kozlowski, 
where | hope we will find shared practice innovations influencing the relevance of our research, at least 
as much as research stands to impact practice. Another translation effort comes in The Bridge, a series 
of excellent T/P columns dedicated to the science-practice interface (there is a Bridge article in this edi- 
tion; with thanks to column editors Mark Poteet and Linda Zugec). The third form of translation is one 
that | participated in, a planned eight-part T/P series called Lost in Translation (by Andrew Collmus and 
Michael Litano). Check these out if you haven’t, as they cover translational issues such as communi- 
cating the value of I-O psychology (October 2016), overcoming critics in applied research (January 2017), 
and how I-O should talk with policymakers and funding agencies (July 2017). There are related videos as 
well, on our SIOP YouTube channel. 


This “translation movement” is something of a paradigm shift for SIOP. As I-O psychologists, we place 
great importance and even pride in the general phenomena and robust findings that come out of our 
work—as we should. We talk happily at the level of constructs and meta-analyses, and because of our 
sophisticated statistical training, we also thrill to the discussion of correlations, reliability coefficients, 
and structural equation models (currently my own heart is racing). However, employees, employers, and 


organizations are problem focused, and they critically require specific examples. They may enjoy benefit 
from scientific generalities but in a way that is not general but rather in a way that might give them 
some form of particular advantage in the workplace. 


Food for thought. We are always working together as Team SIOP, and so please continue to reach out to 
me (foswald@rice.edu) or to Incoming President Talya Bauer (cetb@pdx.edu) if you have any further 
ideas, questions, or expertise along these two fronts that | have covered here. 


It has been an incredible privilege to serve you in all its forms: for example, supporting and working with 
such amazingly talented, knowledgeable, and dedicated SIOP members and the Administrative Office 
(too many of us to single out here); collaborating and taking on various forms of the complexities that l- 
O psychology faces (and will always face) as an evolving field of study and evolving profession; and har- 
nessing and promoting our expertise, so that we have a seat at the table in taking on a wide array of is- 
sues important to society (e.g., ensuring healthy and productive employees and workplaces; developing 
and improving diversity and inclusion efforts; cultivating leaders, careers, and organizations...the list 
goes on, as you know). 


SIOP keeps getting better because of you! 


Lost in Translation: Visually Communicating Validity Evidence 


Michael L. Litano, ! Andrew B. Collmus, ! & Don C. Zhang? 
Old Dominion University’, Louisiana State University” 


In our previous column, we discussed the complexity and nuances of measuring unobservable psycho- 
logical phenomena and the importance of verbally communicating the value of reliability and validity 
evidence to non-l-O psychologists. Our interviews with Fred Oswald, Jeff Jolton, and Don Zhang were 
insightful, impactful, and extremely well-received by the SIOP community. However, we also received 
some feedback from I-O psychology practitioners that emphasized how much more frequently they 
communicate in the forms of charts, figures, and PowerPoint decks than simply in conversation, making 
it difficult to apply the lessons learned from our last column to their current roles. In fact, it is common 
in the business world to have your PowerPoint decks “walk” around the organization after your presen- 
tation, meaning that you must create your presentation to be interpretable and easily understood even 
without talking points to accompany the slides. 


The focus of the current column is just that: How does one effectively communicate validity evidence 
using only visualization? Despite the interesting and relevant topic, we must express how difficult it was 
for us to obtain visualizations that effectively represented validity evidence. We had several academics 
and practitioners volunteer contributions that either (a) we had difficulty understanding, (b) mirrored an 
output you might see in the Journal of Applied Psychology rather than a business meeting, or (c) weren’t 
representative of validity evidence. For example, although some business audiences may generally be 
able to interpret a scatterplot (see Don Zhang’s contribution below), we as a field can and should do a 
better job of taking a validity coefficient and presenting it such that the value can be grasped by anyone. 


The goal of this column is to show how conducting rigorous research on the back end is important, but 
only so far as the evidence/results are effectively communicated to those who decide whether and how 
to act. We hope to demonstrate how one might effectively visualize validity evidence, including how one 
might visually present a rigorous analysis or complex model in such a way that non-l-O audiences can 
understand and appreciate its value. Like previous columns, our hope is that this shift in focus will pro- 
vide anyone who finds themselves “lost in translation” with resources that not only helps the individual 
l-O psychologist, but also builds the awareness and use of I-O psychology in organizations. In the sec- 
tions that follow, Don Zhang, Michael Litano, AJ Thurston, and Daniel Hawthorne each share some of 
the ways that they have effectively presented validity evidence. 


Don C. Zhang 


In a recent conversation with a director of human resources, he (the director) lamented about the low 
prediction of most selection methods. “Only about 50%,” he said, referring to the meta-analytic validi- 
ties reported in the seminal Schmidt and Hunter (1998) paper. Two problems: first, a correlation of 0.50 
translates to 25%—not 50%—of variance explained; second, only 50%? As a field, we are not doing a 
good job of communicating the value of our work when our most treasured selection methods are re- 
ceived with an unimpressed “meh” by people in positions to use them. 


Traditional validity indices such as the correlation and coefficient of determination do not convey the 
practical impact that a selection method has on organizations. Percent of variance explained is not the 


kind of metric business professionals care about, and 25% is not a number that will impress many. But, 
25% is about as high as we can get due to the “validity ceiling” of predictions (Rundquist, 1969). Clearly, 
there is a need to communicate our evidence in a way that resonates with the public. 


There are many alternative graphical and nongraphical displays of validity information (Kuncel & Rigdon, 
2012). But, not all data visualization techniques are created equal. The scatter plot, for example, displays 
the relationship between a predictor and criterion variables on the X and Y axis. The plot below illus- 
trates the relationship between ACT and college GPA, which has a validity of r = .30: not particularly im- 
pressive. At a glance, the relationship depicted in the scatter plot appears equally unconvincing. Yet, 
scatter plots are the primary method for visualizing linear relationships. 


Scatterplot 
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Alternatively, one can 
use an expectancy 
chart (Lawshe & Bolda 
1958). Expectancy 
charts communicates 
the relationship be- 
tween two variables 
(e.g., ACT score and 
GPA) by presenting 
the proportion of the 
sample with score 
above a cut-off crite- 
rion (e.g., GPA above 
3.5) at a given score 
interval on the predic- 


tor (e.g., ACT score between 25 to 27). Believe it or not, the expectancy chart below is generated with 
the same data as the scatter plot. Based on the expectancy chart, one can easily see the predictive effi- 
ciency of the ACT. Students in the top quintile of ACT have approximately 70% chance making the 


Dean’s List 


1.00 > 


Proportion above 3.5 in GPA 
a 
o 


° 
N 
a 


0.00 


o 

N 

a 
r 


18 to 23 


23 to 25 


25 to 27 
ACT 


27 to 29 


29 to 33 


(GPA above 
3.5) at most 
universities, 
whereas stu- 
dents in the 
bottom quin- 
tile of ACT 
have a less 
than 25% 
chance (25% 
takes on a 
whole new 
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Sometimes decision makers are interested in dichotomizing the predictor. For example, an academic ad- 
ministrator might be concerned with choosing an appropriate ACT cut-off score for admitting students 
into the honors college. To visualize the distribution of GPAs between individuals above or below a cut- 
off, we can use an overlapping density plot. The figure below illustrates the distribution of GPAs for stu- 
dents with ACT scores above and below 26. One way to translate the results is with the Common Lan- 
guage Effect Size (CLES, McGraw & Wong, 1992). The CLES for the validity of the ACT can be described 
as, “a randomly chosen person with ACT greater than 26 has a 62% chance of obtaining a higher GPA 
than a random person with ACT less than 26.” 


Density plot for GPA 
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Nontraditional effect size indices and displays are particularly useful when communicating the practical 
value of selection tests. Research from my lab has found that the lay public tend to judge nontraditional 
displays as easier to understand than traditional validity statistics; and more importantly, they had much 
more favorable judgments about selection instruments (e.g., ACTs, structured interviews) when validity 
information was presented with nontraditional graphical displays. 


Despite the benefits of these nontraditional displays of validity, there are—unfortunately—no accessible 
tools for generating these displays. Common statistical packages (e.g., SPSS) do not readily produce 
these alternative displays. In order to facilitate the calculation of nontraditional effect size displays, I’ve 
created a free-to-use web application that allows scholars and practitioners to easily generate and visu- 
alize a variety of nontraditional effect sizes such as expectancy charts, CLES, and binomial effect size dis- 
plays with their own data (https://dczhang.shinyapps.io/expectancyA 


Michael L. Litano 


Entering the applied world, | incorrectly assumed that senior leaders and other organizational decision 
makers would be able to understand and interpret analyses of “people” data that extended beyond sim- 
ple descriptives, such as favorability scores and agree percentages (i.e., percent of employees who 
“agree” or “strongly agree” to a single or set of questions). That’s not to say these leaders didn’t care or 


were incompetent, but because the way that validity evidence tends to be presented is too dense and 
unintuitive. If we assume the perspective of a senior leader with no I-O background and/or a limited un- 
derstanding of statistics, what would we do with correlation coefficients, coefficients of determination, 
or (un)standardized parameter estimates? 


| have conducted numerous interviews and focus groups with senior leaders and three themes generally 
emerge when it comes to digesting and interpreting survey results: they want (a) easily interpretable find- 
ings, (b) to see “trend”—how much scores have improved or declined since the last survey, and (c) what to 
focus on and how to drive meaningful change. As l-O psychologists, we are well-equipped to meet these 
needs, and | have found the best way to present this data is by doing my due diligence as a scientist in the 
background, then presenting findings that are easily interpretable even if it’s not the exact analysis that | con- 
ducted. After all, in people analytics, the goal is to use data-driven approaches to inform people and organi- 
zation-related practices, programs, and processes, not to publish in a peer-review journal. 


Here’s a scenario: Employee engagement is a high-value metric at Organization X. Employees are sur- 
veyed quarterly and each survey includes ten 3-item scales to measure constructs research suggests are 
primary antecedents of engagement (e.g., leader-employee relationship, professional development op- 
portunities, etc.). Here’s one way | have analyzed such data in the background and then simplified the 
presentation of findings for a non-l-O audience. 


Analysis: Every company has their own guidelines for presenting descriptive data (see translation sec- 
tion below), but when analyzing relationships between variables it’s still essential to methodically clean 
data, check assumptions, assess reliability of your measures, and demonstrate construct validity (corre- 
lations are fine with low N, but use confirmatory factor analysis with high N). At this point, we have 
some level of confidence that we are measuring distinct constructs that we are intending to measure. | 
don’t present any of these analyses to leaders outside of my team. 


Because we have already identified employee engagement as the high-value metric of interest (let’s as- 
sume engagement is related to important HR and/or business outcomes), our goal is to be able to tell 
senior leaders with some degree of confidence what they should focus on to drive meaningful change. 
Especially when collected at one point in time, variables in organizational surveys generally share mod- 
erate-strong correlations with one another. Rather than rely on correlations (which do not provide accu- 
racy metrics and consider only the focal relationship of interest) or multiple regression (which can pro- 
vide misleading results when independent variables are highly correlated), | typically use relative im- 
portance analysis. This analysis treats independent variables as orthogonal so that you can more reliably 
assess the unique contributions of each. It easily identifies the most important “predictors” and sets us 
up well to translate. 


Note: Scott Tonidandel & James LeBreton developed free programs for calculating relative weights in 


multiple, multivariate or logistic regression: http://relativeimportance.davidson.edu/ 


Finally, it’s important to consider the magnitude of the findings you are sharing. For example, if | was 
only able to explain 2% variance in engagement, how much of an impact will these variables really have? 
Conversely, in large sample sizes, everything is statistically significant, so it is important examine the 
practical significance (direction and magnitude) of effect sizes. These and other considerations must be 
made as part of your duty as a scientist. 


Translation: Considering what | told you leaders care about, | could present this analysis in two very 
easy-to-understand slides: 


They want easily interpretable findings. Raw mean scores and standard deviations aren’t meaningful to 
most people, so there is a data transformation aspect involved with this step. Each organization has a 
“oreferred” way of communicating results, so you can be more effective if you can cater to their norms. 
In Organization X, managers are used to seeing “agree percentages.” Therefore, | re-code each individ- 
ual survey response variable to be binary (1 = agree or strongly agree on 5-point scale). 


They like to see “trend.” This isn’t validity evidence, nor is it something we often care about as l-O psy- 
chologists. But again, if you can take the leader’s perspective, it’s logical to wonder, “how were my 
scores this survey?”, “how do my scores compare to the organization?”, and “how do my scores com- 
pare to my historical scores?” Generally, Slide 1 answers these questions for the senior leader and pre- 
pares him/her for our “validity story”: 


After remaining relatively stable in Q1 - Q3, Department E engagement fell 11 
pts in Q4 and is 10 pts lower than overall engagement at Organization X 
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They want to know what to focus on and how to drive change. Here, we take the general results of our 
relative importance analysis and translate it for senior leaders. In this analysis, assume we’ve identified 
two variables to have the strongest relationships with employee engagement: leader—employee rela- 
tionship quality and professional development opportunities. 


Even when | am not conducting a random forests or other tree-based analyses, presenting this in a deci- 
sion tree has been one of the more effective ways to communicate this finding to demonstrate what 
employee engagement scores look like in instances where these two variables are rated favorably and 


unfavorably. There are a lot of rules that you—as an analyst, team, and company—have to decide upon 
(e.g., how many tree branches to present, what are the cut-offs for “yes” and “no” decisions, etc.). Once 
those rules are determined, you can use your stats program of preference to codify each of the four de- 
cision tree buckets (high-high, low-low, high-low, low-high) and create a pivot table that generates 
unique engagement scores for each. | always present these in a footnote so that even when the deck 
“walks,” questions about the analysis can be answered. 


As you can see in the chart below, it’s easy to see the percentage of associates who are engaged in each of 
the four scenarios. Again, I’m trying to communicate that these are the two things senior leaders should 
act on—not publish in an academic journal—so | decided to cut the branches off at two layers and codify 
predictors into binary agree scores so that the “yes” and “no” decisions are more easily interpretable (i.e., 
“yes” means an associate “agrees” or “strongly agrees” on average to that set of questions). Slide 2 high- 
lights that 85% of employees that have high-quality relationships with their leaders and have their profes- 
sional development needs met are engaged, compared to only 15% who have neither. 


Employee Engagement Key Drivers 
Note: Data is for Illustrative Purposes Only 
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85% of employees that have high-quality relationships with their manager and are 


provided development opportunities are engaged compared to 15% that have neither 


For those senior leaders who are more statistically and analytically proficient, you can also try to 
present the results of a relative importance analysis like the below: 
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Q4 questions account for 36% variance in engagement; leader-employee relationship & 


development opportunities account for over half of the unique variance 


Daniel R. Hawthorne 


When using this data visualization method, it is important to explain different ways that this shows the 
importance of each variable to the job(s) in question. Green shows the direction of the desired scores 
and the row-height indicates the strength of the influence. A lay person can rapidly look down the 
scores and see what the most important attributes are to a particular job. 


After a lay person understands how the data is visualized, | can use this to get feedback about our data- 
driven regression equations and know if | need to consider adjusting weights based on attributes im- 
portant to the client that we may not have picked up in data collection. We can also use this method to 
have new clients give feedback about transportable solutions and how they might fit with their jobs and, 
ultimately, how they might need to be adjusted to account for organizational differences between clients. 


Additionally, if we’ve conducted a concurrent validation solution, this data visualization method can be 
used to adjust for an organizational change, where an organization wants to select for different employ- 
ees than they currently have. For example, incumbent employees might show a strong association with 
problem solving and ingenuity, and a client might want to shift more to a model where caring and com- 
passion attributes are the strongest weighted predictors. This method of data-visualization would show 
the existing employees’ current organizational profile and make it easy to have a conversation with the 
client about how new employees might be selected to push the organization toward a new organiza- 
tional competency model in line with the organizational change. 
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Data Visualization Legend 


* Behavioral Competencies are made 
up of lower-level personality 
attributes 


* Row-Height for each Attribute was 
calculated from Regression 
Weights; So, taller rows are 
stronger weights 


* Color indicates the direction of 
Regression weights 


¢ Red indicates where poorer 
scores for the job would be 


* Green indicates where better 
scores for the job would be 


Thus, this data visualization example 
gives a lot of information in a compact 
space. Even without any knowledge of 
the two jobs, one can glean a lot of 
information about the similarities and 
differences of each job’s personality 
requirements. 


Additionally, this allows technical 
experts to have detailed conversations 
with laypeople about a very complex 
set of topics (Personality and 
Regression Analysis) with much less 
confusion 
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A.J. Thurston 


Suppose | have an organization with an expensive training program. The program concludes with a final 
test which demonstrates a minimum level of competence and must be passed to graduate (score > 
45). The pass rate is declining. I’ve been asked to develop a screen-out system for their program. Here 


are the validation results of that system. 
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Predicted Criterion Quantiles 
| start by orienting them to the plot, the x-axis shows predicted criterion score quantiles, and the y-axis 
shows the mean actual criterion score for that quantile. l'Il explain what the quantiles are and how they 
were developed, with quantile 1 representing those trainees who scored in the bottom 20% of the pre- 
dicted score, and those in quantile 5 representing those who scored in the top 20%. In describing how 
the quantiles were made, | also emphasize this is a simplification of a continuous scale. 
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Because this is a screen-out procedure, most of the discussion will focus on those in quantile 1. Here, | 
will add back some of the detail that’s been simplified to create the plot; specifically, the 95% confi- 
dence interval for this quantile. In this case, the 95%CI of the actual scores range from ~40-44. Conven- 
iently, | can say anyone who falls into this predicted quantile has a 95% chance of actual failure. 


If the cost of failure is quantified, the discussion continues in dollars, but here it’s important to be con- 
servative. l'Il do so by giving a range of potential savings to temper expectations. The discussion ends by 
implementing a criterion-referenced cut score based on the training needs of the organization, the cost 
of failure, and any other client considerations. 


AJ also shared an app he created in R that he uses when teaching about validity in classification deci- 
sions. We took screenshots to show how it works. The first example shows a predictor—criterion correla- 
tion of .47, with a predicted cut score of 0.3 and actual cut score of 0.5. 
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In the second example, we changed the correlation to .3 and both the predicted and actual cut scores to 0. 
The interactive app automatically updates the accuracy statistics in the column to the left of the graph. 


C O & Secure | htpsy/aithurstonshinyapps.io/ciessificationsccu'acy xe ean Dose: 
Classification Accuracy App Documeniston 
Classification Properties: ‘True Positives: 278 Distribution Properties: 
False Negatives: 190 
s s 
Correlation Tree ves: 329 Sample Sze 
a [°>] ' False Positives: 199 reee 
(Be A 
à Overall Accuracy: 80.70% Pr 
Predicted Cut Score 95% Cl Lower Limit 57.59% 
95% Cl Upper Lint. 63.74% 0 
0 No internaten Rate 5940% © 
Vator [Ace > NIR] O € Predicted SD 
Actual Cut Score g 
1 
7 Kaçsa:021 6 
Ucnema’s Test P. Value 0.92 ct 
2 Actual Mean 
Sensitivity: 58.40% o 
i Specificity: 62.70% % 
pins ond Pos Prod Vaha 5877% iat 
— Neg Pred Vetus 42.43% 
Prevalence: 47 60% 1 
Detection Rate 27 80% 
Detection Prevalence: 47 30% 
Balanced Accuracy 60.59% 
@AJThurston 


Precision: 5877% 
Reest saan% 
Fx 092 


-1 0 1 
Predicted Outcome 


11 


Summary 


This column was the second in our series on translating validity evidence in applied settings. Whereas 
last time we asked I-Os for verbal translation examples, this time we asked how I-Os use visualizations to 
help explain these concepts. To that end, several practitioners shared some of the creative ways they 
simplify validity evidence for nonacademic audiences. We are extremely grateful for these contributions 
and hope that you will be able to use these resources to craft better, clearer, and more impactful visual 
representations of research findings. 


We generally feel that l-Os excel at carefully designing and conducting applied research, then methodi- 
cally cleaning and analyzing the data to uncover meaningful and impactful results. However, scientific 
rigor means nothing if decision-makers do not understand the results and the benefits and conse- 
quences of acting on results versus inaction. With this theme in mind, we’d like to close our Lost in 
Translation series with a rather critical and possibility controversial assessment of our field. This may 
seem like an odd critique, but we believe that generally, I-O psychology graduate programs are not do- 
ing an adequate job of preparing students to be effective communicators of our science and ambassa- 
dors of the field. Despite claiming to be trained in the scientist—practitioner model, our formal education 
asymmetrically emphasizes traditional academic science communication, while virtually ignoring the 
burgeoning and crucial aspects of public science communication (i.e., Sci-comm). That is not a bad thing! 
But it does help our inspiration for this series come full circle. Maybe graduate programs aren’t currently 
equipped to teach the translation of science into practice. 


Internships and consulting opportunities certainly exist for students in some programs, but perhaps a 
part of the solution to maintaining our relevance is to formally incorporate “translation” into the gradu- 
ate curriculum. Intuitively, academics seem best-suited to prepare students for academic careers: they 
are scientists who are incentivized by publishing new theory and rigorous research studies in peer-re- 
viewed journals with an academic audience. Wouldn’t it make sense to hire practitioners as adjunct pro- 
fessors—not to teach core I-O classes but rather to advise how the complex topics we learn function and 
are talked about in businesses? 


For a field that traces its roots back to the late-1800s and whose flagship journal recently celebrated its 
centennial, we are woefully irrelevant to the majority of HR and business professionals (Rose, McCune, 
Spencer, Rupprecht, & Drogan, 2013). One of the most popular and provocative TIP articles of 2017 bla- 
tantly questioned whether I-O psychology has lost its way and identified other fields that are becoming 
better recognized than we are for the things that l-Os do best (Ones, Kaiser, Chamorro-Premuzic, & 
Svensson, 2017). We believe that our generally inability to effectively translate research findings into 
simple and actionable knowledge for organizational decision makers is a momentous impediment to our 
field’s continued relevance. 


Many scientists are now using R, Python, and other languages to create custom and flexible visualization 
tools. Science communication continues to emerge as an independent field as the scientific community 
and general public realize the criticality and shared goals of open-source accessible science—this in- 
cludes communicating complex research in a way that can be digested by the general public. For the 
sake of our field’s effectiveness and relevance, it is time for I-O to get with the program. We hope that 
graduate programs will take note and better prepare I-Os to effectively communicate science to a larger, 
more diverse, and nonacademic audience. 
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In addition to the contributions above, we’d also like to thank Evan Sinar for pointing us toward several 
validity and general data visualization resources that we found to be particularly useful: 


Visualization pan Author(s)/creator(s 
Description (s)/ (s)/ 
resource host(s) 
Dynamic analytics platform that allows visual explora- 
Graphical tion of univariate and bivariate scatterplot matrices, Louis Tay, Scott Parrigon, 
descriptives group means, moderator analyses, & regression as- & James LeBreton 
sumption checks 
Seeing theory A visual introduction to probability and statistics Daniel Kunin & team 
An interactive visualization tool that helps with the in- 
Interpreting terpretation of correlations, Cohen’s d effect size, null . 
: ae A f . : Kristoffer Magnusson 
correlations hypothesis significance testing, and interpreting confi- 
dence intervals 
f on A guest post on Stephanie Evergreen’s website that fo- William Faulkner, Joao 
Visualizing re : . SA ; 
E cuses on guidelines for effective visualization of regres- Martinho, & Heather 
regression sion Muntzer 
An ongoing compilation of key data visualization links Evan Sinar, Eric Do- 
Show me the p aa 
daia and resources originally developed for a SIOP 2016 versberger, & Kristin 
e workshop Charles 
An interactive app that allows you to upload your data, 
Alternative no pp 4 5 y > 
: set criterion cut-offs and other parameters, and visual- 
effect size . ; Don Zhang 
Eu a. ize expectancy charts, density plots, and common lan- 
calculator 


guage effect size indices 
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who contributed to our columns, we’d like to express our sincerest gratitude to Tara Behrend for trusting 
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What’s Next for Lost in Translation? 


This is last column in our recurring Lost in Translation Series. We are going to take some time to consider 
how we might take Lost in Translation to the next level. If you just can’t get enough, please consider at- 
tending our SIOP 2018 Executive Board Session (I-O Value [No Longer] Lost in Translation 
http://www.siop.org/Conferences/18con/Regbk/EB_block.aspx). 


Interviewee Biographies 


Don C. Zhang is an Assistant Professor in the Department of Psychology at Louisiana State University. He 
received his PhD from Bowling Green State University. His research focuses on decision making, statisti- 
cal communication, and employee selection. He is particularly interested in why many managers are re- 
luctant to use evidence-based hiring practices such as structured interviews and mechanical data combi- 
nation methods. He can be reached at: dc.zhang1@gmail.com 
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and global-based business opportunities for ETS. He also provides direct client-facing leadership in client 
management and account development through the development and implementation of scalable and 
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AJ Thurston is an l-O Psychology PhD candidate at the University of South Florida. He specializes in as- 
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The Modern App: How Technology Is Advancing Team-Centric Work 


Evan Sinar Tiffany Poeppelman 
DDI LinkedIn 


Over the recent years, influential industry analysts (e.g., Bersin, 2016; Haak, 2017) have repeatedly cited 
the shift to team-based work as a major business trend warranting a fundamentally reshaped approach 
to talent management and organizational structure. These disruptive forces are driving workflows that 
are less often hierarchical from a long-term supervisor and more often lateral among project-based 
teams, and processes that are less often serial and more often parallel using agile methodologies. Addi- 
tional interest in workplace teams is also on the rise within major I-O publications such as the Journal of 
Applied Psychology (Mathieu, Hollenbeck, Knippenberg, & Ilgen, 2017). These trends piqued our interest 
in understanding the major trends across technologies that have shifted the way we work since our 
Modern App review of virtual working in Jan 2015, and to see what is top of mind in today’s practice and 
for today’s researchers. 


In this edition of the Modern App, we focus on the possibilities—and realities—of the technologies that 
enable team-centered work. We being with an overview of the various technologies and tools that are 
rapidly gaining a foothold in the workplace, and we then shift to sections—taking a representative ra- 
ther than a comprehensive view—of what the latest research within and outside our field tells us about 
the intersection of teams and technology for virtual teams, collaboration and team-building tool adop- 
tion, and the measurement of team dynamics, including a preview of relevant sessions at the upcoming 
SIOP 2018 Annual Conference in Chicago. 


Team-Centric Workplace Tech: Reviewing Market Forces and Emerging Offerings 


Borders are no longer barriers with the advances in modern technology. These tools allow employees to 
work from almost anywhere in the world and push the boundaries of virtual work, both well-aligned 
with the pressures of the global business environment. Due to these abundant application opportuni- 
ties, there are no lack of collaboration technologies in the marketplace. 


Executives in global corporations are investing in technologies to improve within- and cross-team workflows 
to boost business performance and ensure collaboration for in-person and virtual teams. Many technologies, 
platforms, and tools are ones that we’ve seen on the market for the last 3+ years, but their product function- 
ality lists have only continued to lengthen as organizational demands drive deeper investment to meet the 
changing needs of the enterprise. These tools often fall into eight general categories, although the applica- 
tions for cloud-based team/group-enablement software are increasing by the day: 


1. Social platforms that allow for real time communication such as Chatter, Yammer, Jive, 
Teams, and Slack 

2. Planning tools for team project task management such as Trello, Monday (formerly 
Dapulse), ProofHub, Smartsheet, and Asana 

3. Organization of notes from group meetings such as Evernote and Notebook 


4. Information repositories to access and store team materials such as Dropbox, Google Drive, 
and Sharepoint 

5. Videoconferencing for collaborators to meet real time or desk share materials such as 
Zoom, BlueJeans, and Skype 

6. Building environments for product development or team coding environments such as 
Google Docs, Igloo, Github, and Codingteam 

7. Group crowdsourcing and curation of content and ideas such as AnswerHub, Coggle.it, Pan- 
dexio, and Spigit 

8. Text to speech capabilities that offer automatic and human transcription for audio and 
video files exchanged across teams like SookenData and Amazon Transcribe 


Other technologies that are already here are supporting the recording day-to-day work practices with 
the intent of coaching team members on their daily work practices. These include recording customer 
calls to provide the right investment in training on core skills such as sales or other support functions. 
Some examples of these manager-to-employee coaching tools include Monet Software, Join.me, WebEx, 
Brainshark, Gong.io, Ambition, and MindTickle. 


Of course, why stop there when the analyses of the calls and recommendations are being requested, 
which is leading companies to use a combination of tools such as web meeting apps, like Zoom and 
Gotomeeting, together with software like ExecVision.io, and Chorus.ai to automatically record and tran- 
scribe calls, and then analyze them for insights. 


The Technological Trends 


Across these technologies and tools, we see four recurring themes. 


First, mobile matters—that is, most collaborative and team-oriented technologies are investing in mo- 
bile access capabilities to ensure people can access these same services on the go. 


Second, collaboration will continue to remain in the cloud— most technology tools for use within teams 
run on the cloud, often requiring steeper investments and larger storage requirements for companies. 


Third, creative communications will be a key feature to access and share team resources being stored 
—many of these tools, and other newsletter features such as MailChimp and Sway, allow companies to 
consolidate messages or display information interactively to improve engagement and responsiveness. 


Finally, data is being collected everywhere and is being leveraged in new creative ways to provide in- 
sights to companies and employees for analyses around work dynamics, collaboration, and development 


opportunities. 


Three Categories of Teams and Technology Research 


Next, we turn to the research base for teams and technology, within and outside of traditional I/O publi- 
cations, centering on three topics: virtual teams, collaboration tool adoption, and new measurement 
possibilities for teams and groups resulting from new technologies. 


Leading and Working in Virtual Teams 


Virtual teamwork appears to be the most-researched facet of the team-technology intersection. In their 
review of 10 years of virtual teams research, Gilson, Maynard, Jones Young, Vartiainen, and Hakonen 
(2015) identify 10 themes, one of which is technology itself. They recognize technology’s role to facili- 
tate communication and monitor performance of virtual teams but raise cautions that technology can 
interfere with high-functioning virtual teaming due to delays and a lack of clarity in intrateam communi- 
cation. On a positive note, however, they review several studies finding that computer-mediated com- 
munication tools can reduce social loafing and between-member status differences, whereas group-ori- 
ented communication tools can strengthen social ties among team members. Notably for future re- 
searchers, Gilson et al. designate modern group communication tools (e.g., document co-creation, 
meeting tools, project management tools) as particularly underresearched components of the virtual 
teams landscape. 


As a sign of work currently underway to represent the current state of I-O research on the topic, virtual 
teaming is also a topic for two SIOP 2018 sessions: Ng, DeChurch, and Contractor (2018) will be present- 
ing the session Information Sharing in Online Teams: How Interventions Improve Information Processing, 
investigating whether “group information processing interventions [...] improve information sharing dur- 
ing online team discussions.” Smoak, Murphy, Moye, and Deere (2018) will present on Working, Leading 
and Learning Virtually: Storytelling and Roundtable Discussion, and “will share stories of enabling virtual 
leaders, learners, and the broader workforce”. 


Turning to the leader’s role in managing successful virtual teams, these teams require leaders who can 
adapt to new forms of collaboration and interaction that can no longer rely on direct interpersonal con- 
tact. This remains a challenging skill for many current leaders. A recent study (DDI, The Conference 
Board, EY, 2018) found that leaders were less confident in their ability to lead virtual teams than any of 
15 other skills rated. Nor are Millennial-generation leaders confident in their ability to excel in this skill: 
they rated themselves significantly lower in leading virtual teams compared to Generation X and Baby 
Boomer leaders. As a result, it’s unlikely that leader proficiency levels will change merely by an infusion 
of younger employees into leader roles: dedicated development in virtual team leadership as a distinct 
leader skill for leaders at all levels and of all generations may be needed. 


Adoption of Workplace Collaboration Tools 


Though a less concentrated base of research compared to virtual teams, research into collaboration tool 
acceptance and employee use is arguably no less important for the organizations investing heavily by 


installing and encouraging adoption of these tools, and seeking resulting ROI. Three studies from the in- 
formation technology discipline provide initial evidence-based guidance about techniques to boost col- 
laboration tool engagement: 


e Maruping and Magni (2015) found that empowered teams were more likely to have higher in- 
tentions and expectations to explore the use of team collaboration tools. 

e Pillet and Carillo (2016) found habit-forming to play a moderating role between perceived ease 
of use and knowledge sharing via the tools. 

e Bayerl, Lauche, and Axtell (2016) examined adoption processes for new team collaboration and 
video-conferencing software, identifying the roles of both attitude and rationale in securing sus- 
tained adoption within and across teams. They conclude by recommending that “organizations 
need to acquire a new mindset, which treats technology changes in collectives not as a one-time 
deployment, but as a process that requires management on a continuous basis” (p. 782). 


These research streams are critical to counteract the headwinds that collaborative tools often face. Suc- 
cessful implementation and positive returns on investment will only come after overcoming passive or 
even active resistance by team members overwhelmed by the demands of an “always on” communica- 
tion cadence (Mankins, 2017). 


Enabling New Forms of Team Measurement, Gathering Richer Team Data 


Technology advances have also propelled the nature of teams research itself by dramatically expanding 
the volume and variety of data that can be gathered about team interactions and group dynamics 
(Carter, Asencio, Wax, DeChurch, & Contractor, 2015). These technologies include wearable sociometric 
badges that track proximity, ambient sound, and other parameters that can be used to code and classify 
interactions among colocated coworkers. Although these devices are not without their measurement 
perils (see Chaffin et al., 2017), when rigorously implemented, they can unlock a rich and incremental 
level of data about team interactions. The work of Kozlowski and colleagues (e.g., Kozlowski, 2015; Ko- 
zlowski, Chao, Chang, & Fernandez, 2015) — targeting one of the highest-stakes teams soon to be in ex- 
istence, the astronauts traveling and working together on missions to Mars — is a model such project. 


Summarizing and Looking Ahead 


With the continued rise of virtual work and further investments into workplace flexibility for employees 
to have options for their work arrangements, we anticipate continued advancement in many types of 
technologies to support team dynamics. By providing an environment that works for everyone, this will 
result in companies continuing to offer multiple locations where employees can work together, and/or a 
mix of remote working days as needed from anywhere in the world. 


As we watch this space, we’re confident that we will see future acquisitions and merger of these techno- 
logical features and tools which will result in limited products that do end to end planning, storage, anal- 
ysis, sharing, and tracking. 


Additionally, with the younger generation entering into work with new ways to work, investments will 
continue to be made around the development and integration of technologies into K-12 programs 
(Boorstin, 2017). We are already seeing schools and programs that are training youth through these 
tools so they can experience team dynamics. 


Although these are just a few snapshots on what is being used within organizations today, what is coming 
out from our own l-O research, and what we might see in the future, we are more excited to see the way 
these data sets and technologies will continue to push the boundaries in the team and technology space. 


Do you have other technology examples you’d add to the list above or ones that you’ve found to be ex- 
tremely effective in your own team work? Please contact us on LinkedIn (Tiffany Poeppelman & Evan 


Sinar) or Twitter (@TRPoeppelman, & @EvanSinar). 
Ideas for future Modern App TIP columns? We are always open to ideas on what you’d like to see covered. 
Last, see below for the full set and topics of our Modern App columns since January 2017: 


2017 Technology Trends: Are I-O Psychologists Prepared? January 2017 
#SIOP17 Program Preview: Technology Roundup for Orlando April 2017 
#SIOP17 Review: Technology Takeaways From Orlando July 2017 

Modern App: Don't Believe (Most of) The Technology Hype October 2017 
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The I-Opener: Earth, Wind, You’re Hired! 
Or: How I Learned to Stop Worrying and Love Helping Small Businesses Using I-O 


Vinay Patel and Steven Toaddy 
Louisiana Tech University 


Thomas Toaddy 
Retired HR Manager 


Okay, so the title of this one is a joke,* but we should explain it briefly before we dive into the real topic here. 


It rubs our sensibilities the wrong way when we hear about hiring practices that are anything but ironclad, 
legally defensible, and well validated. The use of unstructured interviews or clinical decision making or gut 
intuition, instances of nudging the results of a mechanical decision-making process if one doesn’t “like” the 
results, people hiring based on liking the cut of one’s jib — or, as we joked in our title, hiring based on simi- 
larities in musical preferences.” Perhaps it rubs your sensibilities the wrong way as well. 


But throwing a fully validated, ironclad selection system based on an up-to-date and thorough and appro- 
priate work analysis at a situation like a self-employed journalist hiring a personal assistant is simply not 
prudent.? In terms of time, and scalability, and (at least at present) sheer availability of expertise to exe- 
cute appropriate selection practices, our party line neither does nor, we argue, should compel individuals 
in such circumstances to institute full scientifically based, validated selection practices. So what line would 
and should compel in such a situation? How can we use what our field knows to improve selection prac- 
tices in these real-world situations? Indeed, whom must we compel—business owners or ourselves? 


Perhaps you are rolling your eyes at this point—perhaps you are thinking, “Of course we wouldn’t sug- 
gest such practices in such circumstances; instead, we’d suggest X.” Good! That’s where we’re trying to 
bring everyone via this article. Let’s make sure that we know what folks’ big struggles are and that we 
have something to contribute in those situations, though. 


Perhaps you are rolling your eyes at this point. Perhaps you are thinking, “Why would | bother trying to 
help people who can’t afford my typical services?” Here are three reasons: 


e We want to be the go-to folks when it comes to work issues. This means being on everyone’s 
mind, not just the minds of the big guns. 

èe It is a small world—outside and inside of I-O alike. Dismiss anyone at your peril. 

e |f we can do good in the world, let’s, no? 


So we set out upon a two-step process: First, we leaned heavily on our third author’s career’s worth of 
experience in HR to spin up a metaphor for what we were discussing and to guide our questions and se- 
lection of interviewees at both of the following stages. Then, we went to the individual who inspired this 
topic’ and who does not traditionally work with us I-O folks. We sought to capture hiring issues in terms 
that are understandable to us I-O folks whilst honoring the nuances of our respondent’s own language 
(thanks to John for doing so well in speaking both of our languages and thanks again to our third au- 
thor). Then, we pulled out some focal topics and delivered these to some I-O folks with the challenge of 
“Hey, what can we do for this guy and others like him?” Let’s walk through those three steps in turn: 


First, a Metaphor 


As soon as or even before they are aware that landscaping is a deliberate, well-considered practice, most 
people realize that they want to have a beautifully arranged and maintained lawn/garden/ grounds.” Such 
work is not easy, though—it requires both expertise and effort, and doing the job poorly will result in a dis- 
pleasing, sloppily arranged mess of a yard that everyone else will immediately be able to see, eliminating the 
chance that the offending individual will be invited to parties in the future.® In such a position, one has two 
options: learn to landscape, roll up sleeves, get to work; or hire a landscaping firm. If one(’s family?) is large 
enough and wealthy enough, both options are viable—task someone in the family with landscaping, or fire 
off some money to get someone else to come out with a whole team, to sketch out a plan, sell it to the fam- 
ily, and get to (and continue to get to) work. If one cannot afford either approach—for lack of time, money, 
and/or a lawn of adequate size to get bids from landscaping firms at all—one is kind of, as they say, hosed.” 


Many of you may be scratching your heads (not just because of the use of metaphor) at this point, 
though—you are in this latter group, and you do have an impressive lawn, and you aren’t a landscaping 
expert. How did you do it? Perhaps you reached out to your local extension office or to your local mas- 
ter gardeners (or to a friend with similar credentials or to Internet) to furnish (counterfeit?) just the ex- 
pertise that you needed to take care of your own lawn—you don’t know anything about fruit trees but 
you don’t care because you don’t have any, so you just asked about the maple in your front yard. Per- 
haps you banded together with your neighbors to pool enough lawns and thus enough work to warrant 
the attention of a landscaping firm. 


But what about I-O services instead of landscaping? What about high-quality, effective, legally defensi- 
ble hiring practices instead of an attractive lawn? 


Straight to the Heart of the Matter 


We gave John Davis, president of Paul Davis Automation, a ring to further discuss this topic. John’s situa- 
tion has been in the back of the second author’s mind for the past several years as an example of some- 
one whom the second author couldn’t help with all of his fancy I-O tools. After a short conversation that 
was totally proFRESHinal, we dove right in to the matter. 


We started off by asking John how he approaches hiring in his organization. He told us that he ap- 
proaches hiring in a unique way, in that he keeps a list of individuals who he has either met or found on 
Linkedin whom he feels would be a good addition to his organization. When a position opens up, he 
goes through that list to interview them. His interviews consists of having lunch or dinner with the po- 
tential hire about 10 times, and he uses intuition to make his decisions. There are a few reasons he pre- 
fers this method over using formalized tools, and surprisingly, cost isn’t one of them; he would much 
rather pay to get a great hire the first time rather than going back to the drawing board. 


The reason why he isn’t using formalized tools is because they’re simply a foreign concept. Being the 
president of a small business, he’s never had good luck with using tests for hiring. The reason being, be- 
cause of the nature of the business and work, employees are almost required to do a variety of tasks 
that aren’t clearly categorized within one specific job. According to John, most business owners may be 
aware of liability and of other HR-related issues, but they are not aware or familiar with the proper I-O 
techniques used to hire. 


We were grateful for what John shared with us—not because it bodes particularly well for the services 
offered by the field of |-O being of benefit to him, but because we thought that he did a great job of con- 
veying the multiple levels of complexity associated with his position. From where John stands, tests are 
unavailable or incomplete, recruitment practices are deliberate but unsystematic, and he’s looking for a 
better way but hasn’t heard of one yet. What suggestions or comments does our field have for John? 
How can we make I-O services more accessible (and possibly more affordable) to the general public? 
Here are what a few practitioners have to say: 


The Context, the Problem(s) and the Solution(s) 


Our next set of interviewees helped us think about and, in some instances, reframe the problem—both the 
problem that John is facing and the problem that others like John face. Each of them was helpful in brushing 
aside the notion that we always need to do expensive, time-consuming validation work the moment that we 
walk into an organization (or, indeed, ever); they helped us see ways that we could provide right-sized, low- 
cost solutions that would add as much value as possible—as much exposure to and use of the I-O toolkit— 
with as few pain points as possible. These folks gave us much more information that we have room to share 
here, but perhaps the most valuable pieces that we can cover are captured below: 


Points to Consider 


From the conversations with our folks, some of the reservations we had about our field were reinforced. 
Each of our guests shared the same concerns about our profession, and saw it as an opportunity to fix 
the field rather than to just call this state of affairs a weakness. In Neil Morelli’s (head of Selection Sci- 
ence for The Cole Group) experience, there doesn’t seem to be any availability of |-O to small business. 
In fact, he says probably about 90% of small businesses haven’t even heard of I-O. Luke Simmering (Tal- 
ent Solutions consultant at CEB, now Gartner) has similar experiences and says that smaller companies 
don’t have the same advantages that l-Os know about. So in summary, what we have here is an oppor- 
tunity to benefit small companies and even help with legal vulnerability. 


Why do we have this issue in the first place? 

Why is I-O so largely unavailable to small businesses? Kyle Morgan (associate consultant at Aon) tells us 
that one of the reason our field may not be able to help is that the juice isn’t worth the squeeze; it’s ex- 
pensive when it comes to the ROI. He’s not wrong; our services can add up quickly and may not be 
something that is affordable to small business, at least as our services are traditionally packaged and 
sold. Second, the marketing skills of l-Os aren’t all that impressive.® Neil comments on how we as l-Os 
can do the best job when it comes to designing a selection and recruitment system but are not good 
when it comes to marketing—it's just not what we do. This could be another potential reason why I-O 
isn’t well known in the small business community. Because we now know that we are, at least currently, 
expensive (duh) and are terrible at marketing, why are we even bothering ourselves with this, or more 
importantly, why should smaller organizations even bother with l-0? 


Should smaller organizations even bother with 1-0? 

In short—yes (you may skip to the next section). Although larger firms may not be able to provide consulting 
services to small organizations,? boutique firms exist and can do just that, and they do this by providing, in 
some cases, different or different levels of services than larger firms provide. So yes, something can be done 
here. Luke says that there is always a middle ground, and our other experts also agree. Kyle thinks we can 
always add value or give advice. All three of our experts stress the value of having a good/credible relation- 
ship with managers in smaller organizations.”° Building credibility will help managers understand how much 


value we can actually add to their business. Neil says we can’t approach small businesses with our rigid I-O 
steps like we normally would, but we can start by incorporating something small without deviating from core 
l-O principles: data-driven decisions, operationalization, conceptualization, and standardization.“ From 
Luke’s point of view, once we get our foot in the door we can start to shape what managers do to make it 
more I-O. Now that we know that we can truly help small businesses, how should we start? 


What CAN We Do? 


Surprisingly, a lot. Without compromising on our core I-O principles and by creating that trusting and 
credible relationship with these businesses we can: provide better and more qualified employees, re- 
duce adverse impact, and retain good employees. Neil says that we can do all of this by starting to speak 
the language of business—and not just in terms of jargon, but in terms of ROI and strategic HR and ob- 
jective-oriented planning. Without this, we can be hard to understand or, worse, uninteresting and irrel- 
evant. Next, all three of our experts agree that we should start by supplementing whatever the small 
organization already has with an I-O best practice. This shows them how much utility and ROI we can 
provide them. This helps reduce any apprehension they may have towards I-O. Essentially we are help- 
ing them take a step in the right direction by giving them proper methods. 


Some issues that are relevant to small business include: 


e increased turnover; 
e hiring individuals that are required to do more than one job; 
e ending up hiring people that aren’t cut out for the job. 


Luke suggests that these businesses should start to measure competencies that touch different aspects of 
the jobs. Then give candidates realistic cultural expectations and job previews to help reduce turnover. How- 
ever, this is a candidate’s market; managers need to make sure that they give off the best (and accurate) im- 
pression of their company. Similarly, Neil and Kyle recommend selecting employees based on competencies. 
Neil takes it a step further and also recommends that organizations should always link their hiring process to 
overall goals. Look at the organization's needs and hire based on that. I-Os should use experience and stories 
to motivate managers and build social capital. Once we achieve referenceability, we can paint a picture of 
how we were able to help someone in a similar situation. Specifically in Paul Davis Automation’s case, John is 
already aware of |-O—we can begin to push him a little and to translate I-O into his world. He should think 
strategically and look for observable, measurable business outcomes. This gets him away from the purely 
subjective process. As far as recruiting goes, Neil states that there could be a more holistic recruitment strat- 
egy as far as understanding and setting up a talent pipeline: setting up an employee referral program, creat- 
ing ads and thinking about employer brand, and assessing current and previous employees to see what could 
be done better. What do we need to do on the business side for the brand? At the end of the day we can de- 
sign a process that is empirical. This all happens when we step back and relate all systems to the overall goal. 


However, an unaddressed problem still exists: there are more small business than are I-Os.1* How can 
we cater to them all? Should we just put resources out there and let folks apply them? 


Should we automate? 

Well, this is a complex question isn’t it? Kyle believes that encouraging small businesses to talk to us— 
especially if they encounter a problem that they haven’t before—is the best approach. When it comes to 
putting resources out there and letting folks apply them both Kyle and Neil recommend that someone 


with a smidge of technical training should be the ones to use and implement these tools. We can defi- 
nitely productize some tools and methods, but most I-Os might have a problem with the tools not being 
specific enough.*? However, when small businesses approach us with these tools we can, because of the 
social capital we have built, customize based on their needs. Eh. This might take a bit of a cultural shift. 


What next? 

Now that we think that we CAN help small business but are also terrible marketers, what do we do? 
Luke proposed that larger consulting firms can set up a fund, similar to what we see in the government, 
that is specifically used to help small businesses.** Larger companies and clients that employ these con- 
sulting firms can help contribute to this fund. Also pro bono work always helps ;). 


Neil suggests that we should (a) outsource our brand awareness for I-O, dispelling some of the myths 
around I-O and teaching people about what it means to have science around hiring; and (b) create a 
community of I-Os and support groups that can be accessed by small businesses. This basically breaks 
down to a matchmaking game between what we can offer and what businesses need. We can start by 
creating a contact list of l-Os that do consulting with small business. We need some lead-generation or 
demand-generation system in combination with the marketing campaign. Once matches are made, I-Os 
can start off with small projects and then switch to a strict maintenance relationship after that.” 


This particular area of applying I-O to small businesses is rich with opportunities to do some good and to 
help people who could really benefit from what we do. Our experts gave us a nice preview of what we 
can do, but it is our job as an l-O to help spread this knowledge so that the I-O brand can reach its full 
potential — which, in the end, serves all of us.1® 


Notes 


t Incidentally so is the title of this recurring column. 

* Which could be a proxy for race, sex, age, national origin, religion, &c. 

3 Right? 

4 Though, to be clear, he didn’t inspire the title of the article—John doesn’t hire people based on their musical 
preferences (as far as any of us are aware). 

5 As a business or as an individual or whatever — it’s a metaphor. 

At least they probably won’t sue you for an ugly lawn. Probably. 

7 This is a technical landscaping term. We think. 

8 Read: We’re awful at marketing. 

? Precisely how and why this is the case is beyond the scope of this article, but we concluded that this is, at least at 
present, indeed the case. 

10 Well, also in larger organizations — but we’re talking about the smaller ones right now. 

11 All of which can be accomplished on a shoestring budget if one is so inclined. 

12 Well, okay—there are also more large businesses than there are I-Os, and we get along fine. You know what we mean. 

13 Perhaps the biggest area of selection miscommunication/dissatisfaction is around the word “validity,” as in “I 
bought a valid test from a vendor” >_< 

14 This would allow us to expand our approach so that we could do some of the more fancy and valuable but pricy 
I-O activities in those businesses, but we wouldn’t have to—sometimes just having a free consult (on someone 
else’s dime) could be great. 

15 Of course, it’s not going to be this easy, there is the matter of getting people trained up to, you know, DO the things 
that our experts discussed—aligning I-O with business strategy and communicating that to clients, for instance. 

16 |e., humanity. 


Prehire Screening: A Case Study at CVS Health 
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CVS Health 


“The Bridge: Connecting Science and Practice” is a TIP column 
that seeks to help facilitate additional learning and knowledge 
transfer in order to encourage sound, evidence- based prac- 
tice. It can provide academics with an opportunity to discuss 
the potential and/or realized practical implications of their research as well as learn about cutting edge 
practice issues or questions that could inform new research programs or studies. For practitioners, it pro- 
vides opportunities to learn about the latest research findings that could prompt new techniques, solu- 
tions, or services that would benefit the external client community. It also provides practitioners with an 
opportunity to highlight key practice issues, challenges, trends, and so forth that may benefit from addi- 
tional research. In this issue, we profile CVS Health, winner of the 2017 HRM Impact Award for its imple- 
mentation of an evidence-based, prehire screening assessment for call center job applicants. 


Overview of CVS Health 


We at CVS Health were honored to receive the 2017 HRM Impact Award. We truly believe in the sci- 
ence—practitioner model for which we were recognized and hope that others consider this prehire 
screening assessment as a model from which to learn and work. Within this column, we provide a case 
study of how a team of I-O practitioners applied science and common business sense to build and imple- 
ment a prehire screening assessment and showcase its value to the highest levels of the organization. 


CVS Health is the nation’s leading pharmacy innovation company “helping people on their path to better 
health.” These eight simple yet powerful words are more than a mission—it’s our purpose, and it’s what 
guides the company’s 243,000 colleagues as they go about their daily business. 


As a Fortune 10 company with a unique integrated business model, CVS Health is not your traditional 
drug store. We have built a combination of capabilities to work with patients, payers, and providers to 
offer access to lower cost, high quality sites of care and help improve the lives of millions of Americans. 
To provide a sense of the company’s size and scale, last year CVS Health dispensed or managed about 
1.9 billion prescriptions. About half of those were filled at our pharmacy locations and the rest were 
managed by CVS Caremark, which is the company’s leading pharmacy benefits manager (PBM) with 
nearly 90 million plan members. 


Operating at that scale gives us a tremendous opportunity to make a difference in improving the health 
of our patients, communities, and colleagues. But it can also challenge us to compete for the talent we 
need to sustain our operations and thrive in today’s dynamic economy. Our strategic priority to develop 
a strong pipeline of future talent led us to identify an opportunity to enhance our hiring practices at our 
customer care call centers. As we share methodology and outcomes, we want to highlight our learning 
on the practitioner side, which emphasized how “bridging” science and practice was primarily about en- 
gaging our business partners early and throughout the project, and demonstrating the value and return 
on investment they would reap. 


Development and Implementation of Customer Care Representatives Pre-Hire Screening Assessment 


Because of the cyclical demands of call center staffing in our industry, we hire Customer Care Representa- 
tives (CCRs) at a high volume during certain seasons of the year. CCRs are often the front line to our patients, 
payers, and providers when they need to communicate with our PBM. Our particular challenges were: 


a. We interviewed most applicants, creating a slow and burdensome process for applicants and 
hiring managers. 

b. We based hiring decisions on resumes and interview performance, but this didn’t guarantee 
that new hires had the required technical skills (e.g., ability to toggle and type) while providing 
excellent customer service. 

c. Our applicants are also our customers, so we constantly strive to maintain a positive perception 
of the CVS Health brand in the marketplace. 


Thus, the goals of our intervention were to improve and streamline the existing selection process while 
providing an engaging candidate experience and increasing new hire performance. 


Applying the Science of Psychology to our Screening Solution 


We adhere to best practices as prescribed by the Uniform Guidelines on Employee Selection Procedures 
(EEOC, 1978) and the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999), 
and develop job-related assessment content. This involves: 


e Conducting extensive job analysis to ensure content validity 

e Using a battery of measures, such as situational judgment, high fidelity work simulations, and 
biodata to measure a range of technical competencies and work fit 

e = Including a realistic job preview to engage applicants and provide additional context so they 
could evaluate job fit (Kang & Yu, 2014) 

e Conducting a concurrent validity study to refine content and develop scoring algorithms that 
maximize validity and minimize subgroup differences 

e Pilot testing assessments to ensure they function as intended 

e Continuous monitoring of results 


After the CCR assessment had been in place for a year, we examined its impact on administrative effi- 
ciency and quality of hire. We had several robust metrics that were available before and after imple- 
mentation, allowing us to apply a pre-post quasi-experimental design comparing new hires sans assess- 
ment to new hires with an assessment. It was also impactful to calculate the hours and dollars saved by 
automating the first stage of screening, given that prior to implementation, we had a high touch process 
to screen all applicants. For more detailed information about our methodology and the significant gains 
in effectiveness and efficiency, see our HRM Impact Award profile. 


Bridging Science and Practice 


Before delving into how we spanned the bridge, we would like to highlight the advantages of conducting re- 
search in the business world. First, our work directly impacts business outcomes. Research conducted with real 
incumbents and job applicants may yield critical information that confirms or enriches research findings con- 
ducted in controlled settings. In addition, as we have found at CVS Health, the inarguably positive results of suc- 
cessfully implemented assessments set a precedent and facilitate future opportunities for similar projects. 


Still, even a rigorously developed and vetted screening assessment requires significant commitment 
from leaders who are ultimately focused on running their business. We identified three primary meth- 
ods of impressing on leaders the importance and value of our work: evidence-based science, project 
management, and stakeholder alignment. We’ve shared our evidence-based approach, and for project 
management practices we refer you to information shared by previous HRM Impact Award winners, 
such as Jack-in-the-Box (Schiemann & Seibert, 2017) and Bank of America (Niedle & Littrell, 2016). We 
outline best practices for stakeholder alignment below, in the hope that they provide take aways for 
early-career l-O practitioners and students on how to maintain stakeholder enthusiasm. 


1. Identify your stakeholders. It is critical to know who your stakeholders are and understand their in- 
terests and preferences related to the resources you are about to request and the impact of your pro- 
ject on their responsibilities. The most obvious stakeholders are operational leaders. Below are exam- 
ples of other stakeholder groups who have been integral to the success of the CCR project: 


a. Talent acquisition (TA): Prehire screening assessments are resources for and part of the ac- 
quisition process. We share with TA the goals of engaging applicants early in the process and 
building a high quality workforce. At the same time, there is an inherent tension between 
our goals of both screening out applicants and screening them in, as quickly as possible. 

b. Human resources business partners (HRBPs): HRBPs are the practitioner’s best friends. They 
help us reach operational leaders on whom we rely for resources (e.g., job analysis partici- 
pants, content reviewers, raters). Furthermore, they are often the first colleagues that hiring 
managers turn to when they have questions (or doubts) about the tool or the process. 

c. Vendors: We want work done well and economically. Vendors need to adhere to science 
and regulatory requirements and best practices. Although there is not just one right way to 
achieve this objective, it is important to identify the right blend of efficiency and integrity. It 
is also important to maintain transparency across multiple vendors, especially regarding 
progress and risks that might impact their work. 

d. Applicants: Job applicants are our customers and even if they do not receive—or choose to 
accept—an opportunity, we want to leave them with a positive impression of our brand. 


2. Never stop working for stakeholder buy-in. Obtaining buy-in is not just an up-front activity. It re- 
quires continuous work from start to finish of a project and well past implementation. As new col- 
leagues enter hiring leader roles, as tenured colleagues’ familiarity with the original implementation 
ebbs over time, and as applicant volume fluctuates (creating pressure to fill roles), practitioners have to 
reiterate why we use an assessment and how it brings value. The subsequent best practices are ap- 
proaches we found valuable to obtain buy-in for the CCR assessment. 


3. Manage expectations. The development of a customized, valid assessment is not a speedy solution. Trans- 
parency and communication is critical and the first and most important information for stakeholders is: 


e A realistic timeline; 
e Arelatively accurate description of the resources required; and 
e Assurance that significant resources up front lead to significant gains in the long term. 


Prior to launching, we shared our learnings from similar projects in other areas of our business and from 
stories shared by similar organizations. It was also helpful not just to tell them how we leverage technol- 
ogy but to show them examples of engaging, multimedia assessments. 


4. Collect, incorporate, and showcase input from the business in every facet of development, valida- 
tion, and implementation. Conversations with and feedback from the business were critical to the suc- 
cess of this assessment. For example: 


e We convened a steering committee with leaders from the business, from HR, and TA. We docu- 
mented discussions and key decisions made by this committee. 

e Subject matter experts (SMEs) from the business informed the work by describing work require- 
ments, what differentiates successful from unsuccessful new hires, and the rewards and chal- 
lenges of the job. This was resource intensive but essential to creating content that engaged 
applicants and resonated with our stakeholders. 

e Managers reviewed the draft content and provided feedback before it was finalized. 


Post implementation, this has helped us reinforce how the assessment is uniquely suited to CVS Health. 
Two years later, new managers continue to ask “what the assessment is about? How do | know it 
works?” We are able to tie content back to the contributions of SMEs most salient to the person asking 
the question (e.g., trainers in their location). We can also share exactly how a leader’s feedback in- 
formed a change in direction, and provide the supporting rationale. 


5. Demonstrate the value in stakeholder language. Sometimes, the biggest learning curve after leaving 
school is handling the lack of enthusiasm from business leaders over phrases like “criterion-related validity” 
and “statistical reliability.” To be fair, this is not always the case. Leaders are metrics-oriented and appreciate 
significant results. More typically, our stakeholders are interested in the following types of information: 


e What does it measure? Does it measure everything | need? 

e Isit really improving the quality of our new hires? What is the percent increase in quality [or 
productivity]? 

e = How is it reducing turnover? 

e Is it too hard [too easy]? 


We can describe how the content was created and refined based on SME input; for example, referring 
back to the scenarios gathered during the job analysis to show how the tool captures not only job re- 
quirements but the context of work. 


All the validity and outcome data is available and we meet periodically with leaders to address their 
questions. But, it is critical to describe data in such a way that resonates with leaders: how the assess- 
ment differentiates job applicants and leads to improved operational outcomes. 


Of course, as our performance metrics partners at Mattersight noted, data are great but only as good as 
it is meaningful to the people with whom we share it. It’s been important to understand the audience’s 
comfort level with data visualization and to tailor the message accordingly. This applies when sharing 
data with other analysts as well. As analysts working closely with data, it is easy to forget that it is not 
intuitive to third parties. 


As our assessment partners at Shaker noted, the administrative efficiency gained by simply adding an as- 
sessment to a hiring funnel gets little attention in the literature relative to demonstrating posthire impact 
on performance and retention. The large immediate and measurable result realized in the first year of ad- 


ministration was the reduction in time spent sorting through less qualified candidates. Although the litera- 
ture largely ignores the practical utility and ROI this provides to organizations using assessments as an 
early step in their process, it receives heavy attention from the highest levels of the organization. 


Summary 


Our biggest learning from the CCR assessment project is that building the bridge involves engaging 
stakeholders at the beginning and continuously working to ensure their buy-in for the long term. We did 
this by identifying who our stakeholders are, understanding their expectations and how they operation- 
alize ROI, identifying business metrics and outcomes that are important to them, and translating empiri- 
cally based results into metrics that resonate with them. At the end of the day, operational leaders may 
not be swayed by our adherence to the Uniform Guidelines but may be more influenced when you can 
say “we literally saved 3 years’ worth of recruiter time in the first six months of using this tool.” 


We won minds by showing data that the assessment saved time, increased training to proficiency, bene- 
fited overall performance on measurable outcomes, and provided an informative and engaging candi- 
date experience. We won hearts by showcasing the experience once the assessment was developed so 
our stakeholders could experience for themselves the custom nature of it and the way in which it is 
deeply contextualized to our business. We continue to learn more and better ways to carry out our 
work, but so far, we’ve found that the remaining gaps in that bridge can be smoothed out with strong 
project management and clear, two-way communication with stakeholders. 


Calling Potential Contributors to “The Bridge: Connecting Science and Practice” 


As outlined in Poteet, Zugec, and Wallace (2016), the TIP Editorial Board and Professional Practice Com- 
mittee continue to have oversight and review responsibility for this column. We are happy to announce 
that, starting April 2018, Kimberly Adams (kadams6006@gmail.com) and Stephanie Zajac 
(szajac@houstonmethodist.org) will be assuming editorial duties for this column. We invite interested 
potential contributors to contact them directly with ideas for columns. 


Note 


1 We offer special thanks to Gabriel Lopez-Rivas, CVS Health, Daly Vaughn and Carter Gibson, Shaker International, 
and Scott Inns, Mattersight for their work on the assessment project and contributions to best practices and les- 
sons learned in this article. 
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On the Legal Front: You Say You Want a Revolution 
Rich Tonowski 


The opinions expressed in this article are those of the author and not necessarily those of any govern- 
ment agency. The article should not be construed as legal advice. 


But is the revolution, if you got one, the one you wanted or expected? 


The hottest news is the #MeToo revolution. It is especially remarkable because it was not based in any 
new legal promulgation but by the rise of consensus that sexual harassment in the workplace really is 
unacceptable. EEOC can claim it was on the leading edge with its report on harassment, although, as the 
report acknowledges, there has been previous scholarship on the problem. In particular, that scholar- 
ship has highlighted ineffectual efforts to prevent the problem from occurring, not just responding when 
it happens. Getting control of a dysfunctional organizational culture, well within the I-O bailiwick, has 
been picked up by attorneys in the advice they give corporate clients. Some commentators have called 
for “big picture” strategy that seeks to deal with cultures that may support forms of discrimination other 
than harassment, such as pay inequity. 


The “big picture” approach is in contrast to criticism that has appeared in the legal and sociological liter- 
ature, and was taken up by the EEOC report. The “small picture” approach’ is seen as ascribing discrimi- 
nation solely to “rogue” actors who act against the organization’s interests in their harassment. The or- 
ganization’s response is limited to policy defining harassment as wrong and training on the policy, with 
procedures to report violations of that policy and thus limit the organization’s liability. Some advocates 
of cultural solutions would go so far as to make the organizations strictly liable for all harassment, 
whether management knew about it or not. 


As mentioned in previous Legal Front articles, there seems to be fertile ground for both research and 
practice regarding harassment and lesser forms of negative treatment, termed microaggressions and 
microinequities. Dealing with these is not easy. What constitutes legally recognized harassment can be 
fuzzy, involving considerations of severity and persistence.” The microinequity concept further compli- 
cates the matter, because the perpetrator may be acting without malice or even awareness that the ac- 
tion is offensive. 


The potential of psychology and allied sciences has percolated into business strategy discussions. A re- 
cent podcast (GUntner, Smith, & Sperling, 2018) mentions efforts at debiasing decision making; EEO bias 
is one area, but the broader context is dealing with biases in all organizational decision making. l-Os will 
readily notice here some of the biases that come up with conducting interviews and other assessments 
requiring human judgment. The approach advocated here is “nudging” people along the path to better 
decisions rather than heavy-handed policing of the workplace. That the bias issue might be framed ina 
wider context beyond EEO just might help to get it taken more seriously. 


It has been noted 
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ual harassment 
charges received by 
EEOC have been in 
decline, as has the percentage of “cause” findings. It remains to be seen if women now are more in- 
clined to confront their harassers.? 


Bad behavior outside Title VII time limits for filing a discrimination charge likely are not subject to other 
individual victim action. But an interesting side effect is suits filed by investors who take a loss because 
the stock of the implicated corporations falls; the allegation is that the boards of those corporations 
knew about the problem and did not discharge their duties to protect the corporation (Prial, 2018). 
Then there was the Trump Administration’s expected revolution, with much anticipation—both fear and 
hope—as to what was in store for EEO legal enforcement. The fear was that there would be a full-scale 
assault on disparate impact theory. The hope was that there would be less regulation seen as mindlessly 
burdensome, and the enforcement agencies would be less adventuresome in formulating new theories 
for hounding employers. The past year has little to show, either way. We had two federal agencies with 
opposing arguments as to whether discrimination based on sexual orientation constitutes sex discrimi- 
nation under current law. There is more on this below. Meanwhile, EEOC awaits confirmation of two of 
the five commissioners, Janet Dhillon (Chair-designate) and Daniel Gade. Current Commissioner Chai 
Feldblum (Democrat) has been nominated for a second term. Democrats have a current 2-to-1 majority. 
There has been no nomination for General Counsel, the official who heads litigation against employers. 
One unsurprising development was suspension of plans to require salary information on the annual 
EEO-1 workforce demographics report. The Office of Management and Budget (OMB) called a halt be- 
cause, as some employers have maintained since the proposal was announced, the reporting require- 
ments may be unduly burdensome. Advocacy groups have sued, pointing out that OMB under the previ- 
ous administration had approved the proposal. The Department of Justice (DOJ), defending the suspen- 
sion, is arguing that no final action has been taken warranting judicial attention, and the groups do not 
have a claim to requiring the government to collect the data for them. 


Nonetheless, there have been changes in EEOC activity, albeit not dramatic. For fiscal year (FY) 2017,* 
compared to the previous FY, charge receipts were down to 84,259 from 91,503; “cause” findings were 
slightly down (2.9%), and “no cause” findings were slightly up (70.2%). “Merit” (substantive rather than 
procedural) suits filed by the agency were notably up (184 vs. 86); monetary benefits were up slightly, 
$348.0M versus $355.6M, excluding litigation benefits. Those were down, $42.4M versus $52.2M. As 
has been the trend in recent years, monetary relief is coming mostly from agreements, not litigation. 
The charge inventory was decreased by 99,109, to 61,621. Congressional critics had faulted the agency 
for too much focus on high-profile cases while the inventory was growing. 
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tion. Notably, some 
changes from the Obama Administration are still sticking regarding contractor employment rules: paid 
sick leave, affirmative action goals for military veterans and people with disabilities, nondiscrimination 
for LGBT people, and pay transparency. So far, there has not been much on addressing regulatory con- 
cerns from employers. Talk about combining OFCCP with EEOC has quieted, at least for now. 


Probably the biggest changes have been with the National Labor Relations Board (NLRB), which likely has 
less impact on l-O practice than the activities of the EEO enforcement agencies. NLRB broke with prece- 
dent on several issues during the Obama Administration (Pedrow & Gordon, 2018); there is now restora- 
tion of the status quo ante on acceptance of “reasonable” settlements even if a party objects, a balancing 
test of the reasonableness of an employer’s rules and workers rights, allowing the employer to change 
workplace rules without agreement by the union if the change is consistent with past practice, overturning 
the establishment of “micro units” for unionization and overturning joint employer status based on poten- 
tial joint control. Critics had seen this as a way to attach the parent company in disputes involving a fran- 
chise, even if the parent did not control the practices in question. But overturning this matter did not stick. 
Conflict of interest concerns were raised regarding former board member who participated in the over- 
turning decision. The board then voted to return, at least temporarily, to the Obama-era policy. More 
change is likely. The board’s General Counsel, Peter Robb, has issued a list of matters in which NLRB field 
offices should consult with headquarters rather than follow Obama-era precedents. 


Another revolution, which some have started to suspect had stalled out, got some invigoration. The much- 
anticipated decision from the federal Second Circuit Court of Appeals (based in New York), whether Title 
VII covers employment discrimination based on sexual orientation as sex discrimination, was delivered on 
February 27. Previously, the Eleventh Circuit (based in Atlanta) said no; the Seventh Circuit (based in Chi- 
cago) initially said no, following court precedent on the matter. However, after this decision by a three- 
judge panel, the entire set of active and non-recused judges reheard the case en banc (legal French, the 
judges “on the bench”) and overturned the precedent to yes. A similar scenario played out in New York, 
with the added drama of EEOC and DOJ being on opposite side of the question. In the end, the result also 
was similar to Chicago’s: sexual orientation discrimination is sex discrimination. 


The case (Zarda v. Altitude Express, Inc., 2018) involves a skydiving instructor allegedly fired for being 
gay. He filed a discrimination suit but did not live to taste victory in this decision. His estate has been 
carrying on the litigation. 


The vote was 10-3 for the decision, but the margin of the decision conceals marked disagreement on 
underlying legal theory. All of the judges, including the dissent, agreed that sexual orientation? discrimi- 
nation was wrong; the legal question was whether it was covered by law and by what rationale was it 
covered, because Title VII itself makes no reference to LGBT discrimination. The majority agreed on the 
proposition that sexual orientation discrimination by its nature had to involve sex discrimination, to 
some degree. But only five of the judges signed off on the opinion of the court, written by Chief Judge 
Katzmann. The others who concurred in the decision filed their own opinions regarding which portions 
of the court’s opinion they endorsed. The endorsement count seems to be as follows: 


e Textual analysis over legislative history: Congress intended to put an end to sex discrimination, even 
though it did not foresee the specifics of sexual harassment, hostile work environment, and same- 
sex harassment. Sexual orientation discrimination fits as well. This argument was endorsed in the 5- 
judge plurality opinion of the court, 2 concurrences, and 1 who agreed with the general principle 
that sexual orientation discrimination was sex discrimination (and not with the other arguments), 
for a total of 8. 

e Comparative test: If the person charging discrimination were of a different sex in the same situation, 
would there have been discrimination? If so, then sex must be a factor. This was endorsed by five 
judges. The court’s opinion indicated that this endorsement did not invalidate sex-based grooming 
standards or sex-segregated rest rooms. 

e Gender stereotype: Sexual discrimination can involve enforcement of what the employer thinks is 
appropriate behavior for the sexes; sexual orientation can go against this expectation and so be- 
comes the occasion for discrimination. This was endorsed by six judges. 

e =Associational discrimination: Discrimination can occur by association with someone on the basis of 
protected class. It would be racial discrimination against, say, a White person for association with a 
Black person; the employer has no problem with Whites but does not like Blacks or anybody who 
likes Blacks. The Seventh Circuit held that associational discrimination theory was applicable to all 
Title VII protected classes in its sexual orientation decision and eight judges agreed that it applied to 
sexual orientation here. 

e = Legislative developments: The five-judge plurality dealt with some objections by the dissent and 
found them unconvincing: Congress was aware that there were judicial decisions excluding Title VII 
coverage for sexual orientation when writing the Civil Rights Act of 1991 but did not include sexual 
orientation in the law; Congress has failed to pass a law specifically extending Title VII to sexual ori- 
entation although bills to the effect have been introduced many times; other laws differentiate is- 
sues of sexual orientation from sex. 


Three judges joined in dissent, but here also there was disagreement. The main argument, put forth by 
Judge Lynch, was, “Discrimination against gay men and lesbians is wrong because it denies the dignity 
and equality of gay men and lesbians, and not because, in a purely formal sense, it can be said to treat 
men different from women.” He and two other dissenters argued that neither textual analysis nor legis- 
lative history supported inclusion of sexual orientation, that traditional associational bias had an ele- 
ment of bigotry not found in sexual orientation cases, and sex stereotyping is a separate issue from sex- 
ual orientation. He concluded with a differentiation between constitutional and statutory issues, which 
seems to be a response to a comment in EEOC’s brief that it was strange that two same-sex individuals 
could legally marry (established by the Supreme Court as a constitutional matter) but be fired from their 
jobs the next day for sexual orientation. Judge Lynch pointed out that the Constitution is concerned with 
broad, permanent principles of what the government can and cannot do to citizens; statutes deal with 
specifics that apply to how citizens interact with each other and can change with the times. The issue 


here was that the statute did not change. The two other dissenting judges found this argument unneces- 
sary and did not endorse it. 


What next? One reaction was that, in the short, practical term, not much would change. Sexual orienta- 
tion is protected in a majority of states and by many private employers, so discrimination is already 
banned there. For those that are discriminating, this ruling applies only to New York, Connecticut, and 
Rhode Island (which protect sexual orientation), and so it may be business as usual. There is clearly a cir- 
cuit split on the issue, which is an invitation for the U.S. Supreme Court to intervene. But the Court likes 
to have the legal theories worked out in the lower courts before it issues the final word. The majority in 
this case seems to rally around the notion that sexual orientation discrimination logically involves sex 
discrimination, but after that one judge’s rationale does not necessarily carry much weight with another 
judge. The dissent can point to the fact that various courts historically have given assurance that sexual 
orientation is not covered. Because Congress has shown no indication that it will provide a legislative 
resolution, the dispute falls back on the court, with the question being when will the time be ripe for a 
decision. There is also the matter of religious objection to employing gays and lesbians; this highlights 
that the issue is not a simple variation on sex discrimination. 


Finally, the long-running revolt of the State of Texas (2018) against EEOC’s 2012 guidance on criminal 
background history as a selection procedure got judicial resolution. Texas filed suit to prevent the 
agency from telling the state how to set qualifications for its jobs. EEOC, represented by DOJ, argued 
that its guidance was just guidance; there was no legal action pending against the state on the matter; 
and DOJ, not EEOC, had authority to sue the state for a Title VII violation. Texas responded by adding 
DOJ to the complaint and arguing that the “guidance” was an invitation for meritless private lawsuits. 
The court declined to grant the state’s requests for restraints on EEOC enforcement, with one exception: 
EEOC and DOJ were enjoined from "enforcing the EEOC's interpretation of the Guidance against the 
State of Texas until the EEOC has complied with the notice and comment requirements under the APA 
[Administrative Procedures Act] for promulgating an enforceable substantive rule." As Shea (2018) 
noted, the injunction applies only to Texas and, although other defendants might want to cite to this de- 
cision, another court may rule that guidance really is just guidance and thus not subject to the APA. The 
bigger issue is the deference to be given to agency pronouncements on matters to which the law has 
not given specific regulatory authority. This has been a matter of controversy in recent years, with fed- 
eral agencies issuing guidance and amicus curiae briefs (Latin, “friend of the court;” these are third-party 
opinions on how the law should be interpreted) that some have argued went beyond the law and some- 
times contradicted the agencies’ previous positions. Associate Attorney General Rachel Brand recently 
issued policy that DOJ litigators are not to use guidance material as presumptive indication of a legal vio- 
lation (Tenpas, Etzel, Barker, & Nelson, 2018). On the other hand, presumably employers would want to 
know what an agency’s position is regarding enforcement issues, such as EEOC’s guidance on sexual har- 
assment, awaiting OMB approval as of this writing (Gurrieri, 2018). Meanwhile, EEOC has to decide if it 
will mess with Texas on appeal. 


Notes 


1 This is a simplistic description of a complex situation. For more extensive treatment, see, for example, Green 
(2017). Green also discusses a “cognitive bias revolution” heralded by the Implicit Associations Test. Some may 
question this revolution. 

2 Sperino and Thomas (2017) indicated that courts were ignoring blatant sexual harassment, relying on judicial doc- 
trine on “severe” and “pervasive” not found in Title VII. Shea (2017) noted that sexual harassment itself is not 
found in Title VII and criticized Sperino and Thomas’s views. 


3 EEOC statistics in this article come from www.eeoc.gov/statistics. Besides what EEOC provides, see Maatman, 
DeGroff, Gagnon, & Miller (2017) for narrative and employer-side commentary on the statistics. 

4 This covers October 1, 2016 to September 30, 2017. 

5 The case was about sexual orientation, and so may be not fully apply to transgender matters. But the ink was 
hardly dry on the decision before the Sixth Circuit cited it in its ruling that gender identification was also covered 
by Title VII (EEOC & Stephens, 2018). The court held that “discrimination on the basis of transgender and transi- 
tioning status violates Title VII,” because “it is analytically impossible to fire an employee based on that employee’s 
status as a transgender person without being motivated, at least in part, by the employee’s sex.” In addition, “dis- 
crimination against transgender persons necessarily implicates Title VII’s proscriptions against sex stereotyping.” 
The three-judge panel also was unanimously unimpressed by the Religious Freedom Restoration Act (RFRA) de- 
fense that had worked with the district court, finding that “compliance with Title Vil—without actually assisting or 
facilitating Stephens’s transition efforts—does not amount to an endorsement of Stephens’s views.” Also, “as a 
matter of law . . . a religious claimant cannot rely on customers’ presumed biases to establish a substantial burden 
under RFRA.” EEOC’s enforcement of Title VII was not an undue burden on religious expression. The court also dis- 
agreed with the district court’s finding that the actual substance of the complaint was a dispute over compliance 
with the employer's dress code, noting that merely altering the dress code would not address the discrimination 
Stephens faced because she wanted to represent herself as a woman. Finally, the appellate court remanded back 
to district court the matter of whether there was discrimination regarding payment of a clothing allowance benefit 
available only to male employees. 
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One might think that graduate students in I-O psychology would be quite adept at achieving work-life 
balance. After all, researchers in this field have studied the subject for over 30 years (Greenhaus & Allen, 
2011)! Furthermore, many I-O graduate students read about the work-life interface in their coursework 
and study it for their theses, dissertations, and collaborative research projects. Of course, having a theo- 
retical understanding of work-life balance is one thing, but knowing how to implement those principles 
successfully is something else entirely. In reality, achieving work-life balance is often a struggle for grad- 
uate students, given the rigorous and unstructured nature of graduate schoolwork. Graduate students 
often need to juggle a variety of ongoing assignments and duties (e.g., extensive course readings, inde- 
pendent research, teaching). As a result, they may struggle to put down their work, which can make it 
difficult for them to enjoy their leisure time and take care of household responsibilities. 


In this TIP-Topics column, we explore some of the most common challenges that graduate students in I- 
O Psychology face when it comes to achieving work-life balance, as well as some strategies that they 
can use in order to tackle these challenges head on. In order to shed light on these experiences, the TIP- 
Topics team surveyed current doctoral and master’s students in I-O Psychology. In the first part of the 
survey, students responded to quantitative items about their work schedules, overall levels of work-life 
balance and work-life conflict, and frequencies at which they engage in recovery experiences such as 
psychological detachment and relaxation. In the second part of the survey, they provided qualitative re- 
sponses about the specific challenges that they face when it comes to achieving work-life balance, strat- 
egies that they have used to overcome these challenges, and how their graduate programs have sup- 
ported them in this regard. The TIP-Topics team used content-coding in order to identify common 
themes in students’ responses. Throughout this column, we pull from both key insights from our survey 
and relevant findings in the literature. 


Before we proceed, it should be noted that Aimee Kim and Kelsey Herb published a similar TIP-Topics 
column in 2012 about how I-O students can improve their work-life balance. The authors included a 
wealth of important information about the subject. In order to expand upon their work, we went di- 
rectly to the source by gathering information about graduate students’ actual experiences. 


Breaking Down I-O Graduate Students’ Work Schedules and Work-life Balance 


In order to recruit participants for our study, we contacted faculty members from more than 20 gradu- 
ate programs in industrial-organizational psychology, social-organizational psychology, and organiza- 
tional psychology. Faculty then forwarded the survey link onto their students, many of whom decided to 
participate. Our final sample consisted of 79 doctoral and 42 master’s students. Both samples were ma- 
jority female (62.7% for the doctoral students and 61.5% for master’s). We asked participants to report 
the number of hours per week that they typically spent on work for graduate school (e.g., course read- 
ings, research commitments, teaching requirements). The mean for doctoral students was 49.4 hours 
per week (SD = 14.9), and the mean for master’s students was 35.4 hours per week (SD = 13.1). These 
are quite substantial workloads, as 49.3 hours amounts to slightly more than 6 traditional workdays and 


35.4 hours amounts to about 4 and a half. Results also demonstrated that graduate students accom- 
plished a significant amount of their school work outside of typical business hours (9am-5pm, Monday- 
Friday). For hours worked during weekday evenings, the means were 9.47 for doctoral students (SD = 
7.08) and 12.9 for master’s students (SD = 8.76). For hours worked during weekends, the means were 
8.76 for doctoral students (SD = 4.34) and 12.5 for master’s students (SD = 5.16). 


In order to quantify the extent to which students’ intensive workloads affected their lives outside of 
school, we measured their overall perceptions of work-life balance and work-to-life conflict. Work-life 
balance was measured with four items adapted from the Work-Life Balance Scale (Brough et al., 2014). 
A sample item was “overall, | believe that my work life and nonwork life are balanced.” Participants 
were instructed to only consider their graduate school work when they responded to these items. Work- 
to-life conflict was measured with four items adapted from the Work-Family Conflict Scale (Carlson, 
Kacmar, & Williams, 2000). A sample item was “I have to forgo things that | enjoy due to the amount of 
time | must spend on graduate school responsibilities.” Both scales were scored on a Likert scale from 1 
(strongly disagree) to 5 (strongly agree). Consistent with our expectations, we found that both groups of 
students reported fairly high levels of work-life conflict (M = 3.52, SD = 1.08 for doctoral students and M 
= 4.13, SD = 0.77 for master’s students). Similarly, both groups reported fairly low levels of work-life bal- 
ance (M = 2.84, SD = 0.70 for doctoral students and M = 2.74, SD = 0.64 for master’s students). 


Included in this column you will find a bar I-O Graduate Student Levels of Work-Life Balance and 
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I-O Graduate Student Responses to Work-Life Balance Item 
“Overall, I believe that my work life and non-work life 
are balanced.” 


Strongly Agree 
6.6% 


Strongly Disagree 
b 22.4% 


Neither Agree 
Nor Disagree ~~, 


Specific Challenges in Achieving 
Work-Life Balance 


Overall, quantitative results sug- 
gested that both doctoral and 
master’s students have difficul- 
ties balancing their school re- 
sponsibilities with their life ac- 
tivities and commitments. In or- 
der to probe deeper into the 
reasons why this can be sucha 
significant challenge, we coded 
participants’ responses to quali- 
tative items. We found that the 
majority of students’ work-life 
issues could be categorized by 


9.2% i one of four broad themes: weak 
i temporal boundaries separating 
work and home, weak spatial 
boundaries, low psychological detachment from work, and difficulties balancing graduate school with 
romantic relationships. In the following sections we elaborate on each of these issues, describe strate- 
gies that students can implement for each, and provide suggestions for how graduate programs can pro- 
vide assistance. 


Weak Temporal Boundaries Separating Work and Home 


The vast majority of students mentioned that the general lack of structure in their school schedules 
made it hard for them to separate their work lives from their home lives. Whereas in most jobs it is clear 
as to when the workday begins and ends, such clarity is rare in graduate school. Apart from courses, 
teaching obligations, and lab and program meetings, there are few times each week that are officially 
designated for work. One might think that having this level of scheduling flexibility can be a good thing. 
After all, job autonomy is linked to a variety of positive outcomes, both at work and at home. However, 
many of the students that we surveyed indicated this flexibility was an encumbrance because it made it 
more challenging for them to separate the workday from the “home day.” When there is not a clear de- 
marcation of what constitutes work hours and nonwork hours, work-life boundaries become more “per- 
meable” (Clark, 2000; Desrochers & Sargent, 2004; Edwards & Rothbard, 2000). This makes it easier for 
people to devote more of their time to either work or home responsibilities, which can weaken work- 
life balance (Greenhaus & Allen, 2011). 


Many of the students that we sampled mentioned that their scheduling flexibility simply made it feel as 
though every hour of the day was part of the workday. As a result, many students indicated that they 
checked and responded to emails right up until the time they went to sleep, which took away time from 
leisure and household responsibilities. Others noted that they could not afford to make plans with 
friends and family on weekends because there was always something else that they needed to work on. 
Several also mentioned that not having a set workday made it difficult for them to maintain a healthy 
lifestyle. For example, many often chose to forego sleep, exercise, and cooking because they felt that 
time was really meant for reading articles or working on their independent research. 


Graduate students who face these issues may be able to improve their work-life balance by creating a 
clear boundary between work time and nonwork time. Some students provided great recommendations 
for how to go about this. For example, several mentioned that they create their own “set” work hours, 
so that they need to put down their work once those hours are up. Others indicated that while they are 
not quite as strict with their work overall, they refuse to check their email after 5 or 69m. Faculty mem- 
bers can also help their students by sending them fewer emails in the evenings and on weekends. This 
could make a significant difference, as many students noted that their advisors expected them to an- 
swer emails at all hours and make themselves available at a moment’s notice. Another useful suggestion 
was to wake up on the early side, so that one could have ample time to do work during the day and re- 
lax at night. Many also mentioned that each week they strive to make either Saturday or Sunday an en- 
tirely work-free day. 


Weak Spatial Boundaries Separating Work and Home 


Along the same lines, many students indicated that their issues with work-life balance stemmed from 
weak spatial boundaries between work and home. This is quite sensible, given that graduate students 
generally do a significant amount of their work at home. Research suggests that when people work from 
home, there can sometimes be a lack of clarity between what constitutes one’s home space versus 
one’s work space. Such permeable spatial boundaries can make it more likely that stressors originating 
in one domain will spill over into the other (Clark, 2000; Desrochers & Sargent, 2004; Edwards & Roth- 
bard, 2000). When stressors from either the work or family domain interfere with functioning in the 
other domain, the result is role conflict. Such conflicts can threaten one’s ability to maintain work-life 
balance (Greenhaus & Allen, 2011). 


We found somewhat similar patterns in our graduate student sample. On average, we found that doc- 
toral students completed 44.7% of their work at home and 43.4% of it on campus. The percentages for 
master’s students were nearly identical, as on average they completed 42.8% at home and 43.0% on 
campus. This even split between home and campus is promising, because it suggests that there are at 
least some spatial boundaries. Nevertheless, this is still a significant amount of “homework,” and many 
of the students we surveyed mentioned several ways that this negatively impacted their work-life bal- 
ance. For example, some indicated that there were many distractions at home that made it difficult to 
focus on work, such as chores that needed to get done, the television remote being an arm’s length 
away, and roommates or significant others relaxing on the couch after completing their day at work 
when the graduate student has a paper due the next day. Others mentioned that their home simply felt 
like another office space, which made it difficult for them to put down their work and enjoy their leisure 
time. This also made it seem as though the people they lived with were encroaching upon their work- 
space, which made their work more stressful and occasionally led to them taking this stress out on those 
around them. 


In order to circumvent some of these issues, students noted that they could create designated work- 
spaces in their apartments or homes or spend more time doing work in coffee shops, libraries, and on 
campus. Another good idea would be to be straightforward with roommates, significant others, and 
family members about the need to use the home as a workspace. When there is a respectful agreement, 
this can benefit both parties. Finally, graduate programs can do a better job of providing their students 
with designated workspaces on campus. Although nearly every doctoral student had such an arrange- 
ment, few master’s students did. 


Difficulties With Psychological Detachment 


Another common issue was that even when students were engaged in nonwork activities, they could 
not take their minds off of their work. In the literature, we might refer to this as a lack of psychological 
detachment. Psychological detachment from work refers to the extent to which one mentally disen- 
gages from work during off time (Sonnentag & Fritz, 2007). This can be particularly important for gradu- 
ate students, who face a multitude of work demands day in and day out. If they are unable to psycholog- 
ically detach from work, then these demands will continue to strain students’ well-being and make it dif- 
ficult for them to enjoy their nonwork time. Many students mentioned that it was difficult for them to 
be fully engaged in activities with friends and family because they could not stop thinking about the 
many work assignments that were on the horizon. Others even reported that they felt guilty when they 
decided to relax because of a perceived obligation to spend that time reading for classes or making pro- 
gress on research. We also measured students’ levels of psychological detachment, using the psycholog- 
ical recovery scale (Sonnentag & Fritz, 2007). The results were consistent with students’ qualitative re- 
sponses. Mean scores on the five-point Likert scale (with higher scores indicating more frequent detach- 
ment) were fairly low, at 2.36 for doctoral students (SD = 0.74) and 1.93 for master’s students (SD = 
0.61). 


This can be a difficult problem to address, as changing one’s mental habits is not an easy task. However, 
there are some useful strategies. For example, exercising, doing yoga, and practicing mindfulness can 
help clear one’s head and break free from graduate school thoughts. Although these were the most 
commonly used strategies, students can also psychologically detach by spending their leisure time on 
activities that are particularly engaging or mentally stimulating (e.g., hobbies, new challenges). These 
types of activities can help students recharge their batteries without letting work thoughts creep in. An- 
other strategy is to simply take short breaks throughout the day in order to periodically clear one’s mind 
from work. 


Graduate School and Romantic Relationships 


Last, many students brought up challenges related to being in a romantic relationship while in graduate 
school. The majority of students in our sample (64%) were currently in a romantic relationship. Most of 
these issues seemed to stem from students having drastically different work schedules than their ro- 
mantic partners. For example, many students noted that if their partners worked standard full-time 
jobs, there was an added pressure for them to get their work done during those hours. There was often 
an expectation that once their partner got home, the workday was over and the remaining time was 
meant to be spent together. Several students suggested that this pressure was actually beneficial, as it 
forced them to create a set work schedule for the week (which helped improve spatial work-life bound- 
aries). However, others noted that their partners’ expectations were simply unrealistic, because they 
often needed to work during the evenings in order to stay on track. This could place strain on the rela- 
tionship. A related issue was that students felt that their partners had very little idea what it was like to 
be a graduate student. Evidently, some partners believed that graduate students had tons of free time 
on their hands, because class time takes up such a small fraction of the week. Others apparently did not 
realize how challenging and time intensive research projects can be. Many noted that these types of 
misconceptions were disrespectful and led to arguments about whether one was prioritizing work over 
the relationship. Although there is no “correct” way to handle such issues, having honest conversations 
can go a long way. Graduate students can be forthright about their workloads and work schedules, and 
make it clear that at least on occasion they will need to work in the evening. However, they can also 
make concerted efforts to synchronize their schedules with those of their partners. 


Furthermore, some students lamented that because they had the ability to work from home on a regu- 
lar basis, they were often expected to handle the lion’s share of household duties (e.g., laundry, dishes, 
pet care). Again, being forthright can go a long way. Having an honest dialogue about why this is frus- 
trating may lead to a more equitable distribution of chores. 


Conclusion 


The common threads in this survey revealed that many doctoral and master’s students in I-O psychology 
share concerns about the state of their work-life balance. Our survey revealed that both doctoral and 
master’s students in I-O psychology report fairly low levels of work-life balance and high levels of work- 
life conflict. We found that many of these work-life issues stem from unclear temporal and spatial 
boundaries between work and home, difficulty psychologically detaching from work, and trouble balanc- 
ing graduate school with romantic relationships. Although there is no doubt that these are significant 
challenges, so many of the surveyed students offered clear, clever, and optimistic insights about how to 
combat the difficulties that they face. We hope that by being forthright about these work-life conflicts, 
we can foster open discussion about strategies that can be implemented on the individual and institu- 
tional levels. Conversations like these can provide I-O students with a sense of camaraderie and resolve. 
We are in a unique position to confront our struggles from an insider’s perspective given the nature of 
our academic studies. Our work-life problems may be difficult, but we have the tools and wherewithal 
to solve them together. 
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Crash Course in I-O Technology: A Crash Course in Blockchain 


Richard N. Landers and Andrew B. Collmus 
Old Dominion University 


Blockchain seems to be all the techno-rage these days. It is the technology underlying the cryptocur- 
rency Bitcoin and the many altcoins that have come after it, such as Bitcoin Cash, Litecoin, and 
Ethereum. Blockchain is poised to “disrupt” several industries, and consultancies like Deloitte claim that 


HR disruption is ahead. 


To understand what might be disrupted and if I-O psychology should care, we need to dig a bit into what 
exactly blockchain is, what it does, and what potential it offers. The best way to understand a technol- 
ogy is to recreate it yourself—think about how many years ago you learned ANOVA by creating a sum- 
mary table by hand—so in this article, we will create a small blockchain using R to illustrate just how 
simple the basic concept is. But first, let’s walk through it with words. 


A blockchain is, at its core, a digital ledger. In most cases, these ledgers are public and available on the 
Internet. Each blockchain is a list of permanent and secure information that you want to keep and add 
to over time. Each of these groupings of information is a “block.” In the case of cryptocurrencies, blocks 
contain information about transactions. For example, a Bitcoin ledger contains, among other things, a 
list of sale amounts, buyer IDs, and seller IDs. Each time a new transaction is added to the Bitcoin 
ledger, the ledger grows by one block, and blocks can never be removed. Thus: block-chain. 


If you’re wondering why permanent public digital ledgers would be useful, it is most obvious in the case 
of cryptocurrencies, which is why we hear so much about them these days. There are three major ad- 
vantages. First, because buyer IDs and seller IDs are encrypted, money can change hands without the 
identities of the buyer and seller becoming public. In other words, | know that “user 123” made a pur- 
chase from “user 456,” but | have no immediate way to determine who 123 and 456 actually are. How- 
ever, if john@iopsych.com comes to me and says “I am user 456,” | can verify this easily by encrypting 
“john@iopsych.com”. If my encryption turns up “456,” | have verified this person’s claim. 


Second, because the ledger is public and on the Internet, there is no need for middlemen, which is a ma- 
jor issue in the financial sector. If you have ever tried to send money over state lines, or worse, interna- 
tional borders, you might have noticed that there were a surprising number of organizations involved. 
For example, if | want to send money from the United States to a friend in Germany, | can’t simply “send 
money.” Instead, | would probably need to initiate a wire transfer, which typically carries fees and in- 
volves companies called “brokerage houses.” The most common of these is Western Union. That 
means if | want to send money to my friend, information must be collected, verified, and processed by 
me, my bank, the brokerage house, my friend’s bank, and then my friend: a minimum of five distinct 
steps are involved. The transactions between each of us and our bank, and between each bank and the 
brokerage house, are all under a complex system of local, regional, national, and international laws. By 
using a cryptocurrency, you can bypass literally all of that. At least for now. 


Third, because the ledger is public, it can be verified and tracked by an unlimited number of people and 
organizations at any given moment. If someone tries to change an early block in the chain, the entire 
remainder of the blockchain is suddenly incorrect. 


This combination of security, anonymity, and speed is what has made cryptocurrencies enormously pop- 
ular, because these three characteristics are all quite valuable for financial transactions. But can block- 
chain, the technology that enables cryptocurrencies to even exist, benefit HR and l-O? To understand 
this, let’s first look at blockchain in action. 


Let’s See It in Action 


As described earlier, the best way to understand blockchain is to create one. It’s not as difficult as it 
might seem! Creating a blockchain fundamentally only requires two actions: the creation and encryp- 
tion of blocks. We’ll do this in R. 


First, let’s ensure we have the library we need, which we will use for encryption. 


library(digest ) 


Next, let’s create a sample block so that we can see what goes inside: 


starter_block = list( 
index=0, 
timestamp=Sys.time(), 
data="important information for first block", 
prior_hash="genesis", 
next_hash=NULL 
) 


This block is coded as a list object that contains five pieces of information. The index simply counts 
which block we’re on, starting at zero. The timestamp records when the item was added to the ledger. 
Data contains whatever it is we want to track. 


The next two items, prior_hash and next_hash are the key to blockchain. A hash is the result of encryp- 
tion. There are numerous algorithms available to convert literally any piece of data into a cryptographic 
hash, and the better the algorithm you use, the more difficult it will be to reverse engineer the hash to 
produce whatever created it. For example, try this in the R Console: 


shai("Encrypt me!") 


You should see the following output: 


2a2419070644fecacab078ebfic6c92408elbaef 


The fact that you and | and everyone else get exactly the same answer is the key to hashing; there is 
(theoretically) only thing that will produce that precise hash (i.e., a string containing the text, “Encrypt 
me!”), it took very little time to create the hash, and it’s very difficult to go in the reverse direction (i.e., 
from the hash back to “Encrypt me!”). Thus, good hashes are easy to create, easy to verify, but difficult 
to reverse engineer. 


The hashing algorithm we just used is called SHA-1, which stands for Secure Hash Algorithm 1. These 
days, SHA-1 is not considered secure, because computing power has increased so dramatically over the 
past few years that SHA-1 can be reverse engineered in a reasonable amount of time with a sufficiently 
powerful computer. Thus, modern cryptography typically uses some variant of SHA-2, which comes in 
224, 256, 384, and 512-bit varieties. The most common of these is SHA-256: 


digest("Encrypt me!","sha256" ) 


When you run that, you'll see that the output hash is much longer. In fact, due to its length, that hash 
has 27° possible values. Thus, it is incredibly unlikely that any two strings, any two pictures, any two au- 
dio clips, any two videos, or any two anything, after being hashed, would produce the same SHA-256 
hash. This translates into higher security, because it is extraordinarily difficult to reverse engineer the 
original data from which a SHA-256 hash is created (with modern computers). 


Given this, the security advantages of blockchain come from the fact that each new block is hashed 
along with all previous blocks, and each new hash is stored inside each new block as it is created. Thus, 
if you change the information contained within an early block, all the hashes that come after it will sud- 
denly be incorrect. This prevents people from attempting to rewrite the past within the ledger, because 
it will be immediately obvious to everyone else watching the ledger that something has changed that 
should not have been changed. 


To illustrate, we will create two functions. First, we need a function that hashes a block. In the function 
below, we apply the SHA-256 hashing algorithm to the information contained within a block, then return 
the hash it creates. 


create_block_hash = function(block) { 
to_be_hashed = list( 
block$index, 
block$timestamp, 
block$data, 
block$prior_hash 
) 
next_hash = digest(to_be_hashed, "sha256") 
return(next_hash) 


} 


As you can see, the function takes a block as input, then it hashes four of the pieces of information 
within that block, returning that hash as the output of the function. 


Next, we write the function to create a new block based upon that hash: 


create_new_block = function(prior_block, data_for_new_block) { 
new_block = list(index = prior_block$index + 1, 
timestamp = Sys.time(), 
data=data_for_new_block, 
prior_hash=prior_block$next_hash 


new_block$next_hash = create_block_hash(new_block) 
return(new_block) 


In this function, the previous block in the blockchain plus the new data are used as input to create a new 
block. Then, this data is hashed using the first function | wrote above and included as part of the new 
block. That means the block now contains unencrypted data plus an encrypted hash of those data. 


That’s it! We have the two functions necessary to create a blockchain. To illustrate what’s happening in 
these two functions, try hashing the starter block we created earlier: 


starter_block$next_hash = create_block_hash(starter_block) 
starter_block 


When you run this code, you’ll see that the starter_block list now contains a “next hash,” which is a SHA- 
256 encryption of itself. We can start our blockchain with this newly encrypted block like so: 


blockchain = list(starter_block) 


and we can add new blocks to the blockchain directly: 


blockchain[[2]] = create_new_block(blockchain[[1]], "new info for block 2") 
blockchain[[3]] = create_new_block(blockchain[[2]], "new info for block 3") 
blockchain[[4]] = create_new_block(blockchain[[3]], "new info for block 4") 


Ina “real” blockchain, to be used in the real world, we would want to create code to add blocks like this 
automatically, based upon requests from the people watching the blockchain ledger. But for our toy ex- 
ample here, that just adds unnecessary complexity and is something we can leave to professional soft- 
ware engineers. 


© [3] list [5] List of length 5 
index double [1] 2 
timestamp double (53: POSIXct, POSIXt) 2018-03-04 13:34:50 
data character [1] 'new info for third block' 
prior_hash character [1] 'fc27963ce86861ca00e7cc5dcb2725063f88bfbbe817c08d59822347a266add6' 
next_hash character [1] '9c648cd89c1216f93d2ef57184504db545eae7ba4ab9135ecd10ce620b4e6860' 
© [4] list [5] List of length 5 
index double [1] 3 the same! 
timestamp double (53: POSIXct, POSIXt) 2018-03-04 13:34:51 
data character [1] ‘new info for third block’ 
prior_hash character [1] *9c648cd89c1216f93d2ef57184504db545eae7ba4ab9135ecd10ce620b4e6860' 
next_hash character [1] *9f412e14c82cb09f99e35604385be9b502d434a959bb53ece66014e05b9debdb' 


You can see the outcome of the hashing algorithm by looking at the content of our four-piece block- 
chain. Within each block, the next_hash character vector gives a SHA-256 hash that is also the SHA-256 
hash contained in the prior_hash vector of the next block. 


Put in programming terms, two relationships are therefore always true for any two sequential blocks 
within the chain: 


1. blockchain|[[i]]Snext_hash == blockchain[[i+1]]Sprior_hash 
2. blockchain[[i+1]]Sprior_hash == create_block_hash(blockchain|[[i]]) 


By building a chain of blocks where these two equalities are always true, all blocks within that chain are 
linked together in a permanent sequence. If any block is removed or changed, the hashes will no longer 
be accurate, and this is immediately obvious to anyone watching the ledger; that’s what maintains the 
security of the blockchain. 


To Learn More 


The R code | wrote to create a blockchain here is a simplified version of Datacamp’s exploration of block- 
chain in R, so if you want to dig just a little deeper, or if you want to see the same ideas in Python, | 
would recommend starting with one of those. Once you have a better handle on the core idea, you 
might look at something like Hyperledger, which is a complete framework for creating your own block- 
chain and adds a lot of features that don’t exist in our toy blockchain, like proof of work, decentraliza- 
tion, and a management layer. 


Having said that, frankly, I-O psychologists are never going to need to create a functional production- 
quality blockchain except to satisfy personal curiosity, so for the average I-O, the toy blockchain pre- 
sented here is plenty to play around with. Once you have a handle on the idea of a public-facing ledger 
that contains permanent data records and encrypts previous versions of itself as it is built over time, you 
have the core of the idea. The remaining issue to explore is where this idea might be useful. 


Conclusion 


Understanding where blockchain could be applied to I-O and HR more broadly is right now the million- 
dollar question (literally). There are a host of startups attempting to take advantage of this new tech- 
nology to improve HR. 


We reached out to many I-O, HR and business leaders to get a better feel for what type of questions and 
requests are being made regarding the use of blockchain in talent management. The majority of our re- 
spondents were unwilling to share or had not yet been approached with such questions. However, the 
head of an advanced analytics department in a large technology firm who wished to remain anonymous 
offered the following: 


Blockchain has great potential in HR to speed up the delivery of employee information 
through the systems. Examples would be having all of your medical plans attached to 
your medical history within blockchain such that you can give your doctor the encrypt 
key any time and they would have a full history of you. 


Blockchain could easily help in the onboarding/validation of new employees. Imagine if 
all of your employee docs and required paperwork were on blockchain. The amount of 
time saved from an HR onboarder to go find all relevant information would be dramati- 
cally faster than having to call around to each place to verify documents and history of 
employment. 


Blockchain could eventually impact people in job search and recruiting. If a user has re- 
sume on blockchain, it's verified already and makes the search and hire process faster. 


The most obvious and likely the first to happen will be an employee payment going 
through blockchain. This could save significant time and cost around all the 3rd party 
payers for companies. This would be beneficial for full time, part time, and contract 
workers. 


It could also be an electronic replacement for those countries that need to have a paper 
copy of certain things for record. If those all switched to blockchain that is secure and 
distributed, there would be no need to create all the paper trail, storage facilities, and 
people to manage that paperwork. 


And lastly, it will enable better security and relevance in areas like EMEA and beyond 
which have much stricter laws around global data privacy regulations. 


Deloitte identified several use cases for blockchain in HR. Some of these are more relevant to I-O than 
others. One use case was payroll. Specifically, because many organizations are multinational entities, 
the process of paying their employees must move through several intermediaries as mentioned earlier 
in this article. However, because international money markets are volatile, some intermediaries take ad- 
vantage of exchange-rate fluctuations to profit by optimizing the timing of payouts. Thus, in addition to 
cutting out the middleman, blockchain can be used here to lock in a specific and verifiable exchange 
rate, leading to greater transparency and efficiency in the exchange. This is probably the biggest and 
most obvious advantage to blockchain in HR, although it is not particularly relevant to I-O. 


Another HR use case identified by Deloitte was credential certification. Because job applicants will 
sometimes fudge or outright lie in application materials, blockchain could be used to create and store 
verifiable histories of credentials, such as GPA and degree earned from a university, background check 
status, or any other information used to make personnel decisions. In l-O terms, this ideally increases 
the validity of (verifiable but often unverified) application materials and employment records. Biodata, 
in particular, would benefit from blockchain. 


A third use-case extended the idea of credential verification to include all types of HRIS information, in- 
cluding home addresses and medical records. Thus, anyone needing this information and given the key 
could access that information immediately without the need for technological middlemen. This is at- 
tractive due to the difficulty in reverse engineering modern encryption; theoretically, the people who 
should have access to the data can get access very easily, and the people that shouldn’t, can’t. How- 
ever, it should be noted that today’s “secure encryption” is tomorrow’s data breach, so organizations 
and HR departments would do well to exercise caution when dealing with confidential data regardless of 
the level of encryption. Traditional data security practices are unlikely to disappear, but they are likely 
to change shape. For I-O, the greatest advantage to blockchain of this type would be a more consistent 
way to access HR information when needed; anyone who has tried to integrate data from multiple HR 
data systems and thought, “Why does this have to be so difficult?” has already wished for blockchain 
without realizing it. 


In summary, blockchain is a popular new technology that allows for decentralized, encrypted, and verifi- 
able exchange of information. Each new block takes the existing information including an encrypted 


hash of that information, adds new information and a timestamp, then encrypts it all again before send- 
ing to the next block. As the latest buzzword in technology, blockchain will certainly be brought up to I-O 
practitioners as organizations and HR departments follow trends and strive for competitive advantage. 
We’ve identified some potential use-cases for HR. As with all new technologies, it is important to con- 
sider whether blockchain is necessary and actually adds value to an existing process. There will undoubt- 
edly be new products, vendors, and services peddling blockchain in HR. We hope that with an increased 
understanding of what blockchain is and how it might be used, our readership is better able to navigate 
these claims and differentiate between innovative useful products and those capitalizing on buzz. 


That’s it for the eighth edition of Crash Course! If you have any questions, suggestions, or recommenda- 
tions regarding blockchain or Crash Course, Richard would love to hear from you (rnlanders@tntlab.org; 
@rnlanders). If you have questions about blockchain in particular, my guest coauthor this week, Andrew 
Collmus, would be particularly happy to help (acollmus@odu.edu; @AndrewCollmus). 


So You Have Tenure: What Comes Next? 


Allison S. Gabriel 
University of Arizona 


Chu-Hsiang (Daisy) Chang 
Michigan State University 


Russell E. Johnson 
Michigan State University 


Christopher C. Rosen 
University of Arkansas 


After focusing on nothing but researching and teaching since entering the tenure track, one of the fun 
pieces—| hope—posttenure will be figuring out how to shift my time around. Part of this shift will ideally 
be to regain a bit of work-life balance that has vanished since | started my PhD program 10 (!) years ago. 
| admit that this vanishing act was intentional, and | created it because | love my job, and that means 
work doesn’t always feel like “work.” However, | recently read this article by Katerina Bodovski on The 
Chronicle of Higher Education entitled “Why | Collapsed on the Job,” and many of the points hit home 
and fit with my own struggles that | have detailed here. So, goal #1 posttenure is going to be making 
small changes that add up in a big way: trying to minimize the amount of e-mailing/working | do on the 
weekends, leaving work at work as much as possible during the week, and maybe (just maybe) leaving 
my work computer at home whenever we leave Tucson for more than a day. Of course, as Mike and | 
have talked about these shifts we both want to make (he’s an academic, too, teaching math at UA), we 
know that there will be exceptions when deadlines and exams come up. But, we both have agreed that 
the “life” part of work-life balance needs to roll more to the forefront. 


Besides rebalancing things, the other conversation that is dominating our household these days is: What 
comes next? Although we’ve talked a lot about new classes | might like to teach or how | want to change my 
approach to PhD advising, the one piece that is entirely new compared to discussions pretenure is what types 
of service outside of UA | see myself getting involved in. I’ve been fortunate to have some really positive ser- 
vice-related experiences as a junior faculty member, from co-coordinating the PhD program at UA with Na- 
than Podsakoff, to being on a few journal editorial boards, and, more recently, to becoming increasingly in- 
volved at AOM and SIOP. But, despite these experiences, there is a lot of uncharted territory out there. Ra- 
ther than guess about what it might be like to try different service roles, | reached out to three friends and 
fellow Akron alumni—Daisy Chang, Russ Johnson, and Chris Rosen—to talk about working for NSF, editing at 
top journals, and entering leadership tracks in our professional organizations. 


Daisy Chang: On Becoming a Program Director at NSF 


One of the major benefits of being posttenure is that it gives you more autonomy to pursue projects and 
engage in roles that you are interested in. In my case, | was looking to do something new and challenging. | 
have been researching and teaching since | started graduate school in 2000, and | felt like | was in a cycle 
dictated by academic calendars, conference dates, and other deadlines for submissions and revisions. 
Don’t get me wrong, | still very much enjoy interacting with my students, battling the reviewers, and hav- 
ing the summers off. But | was looking to expand my horizon a little bit. This is when the opportunity to 
take on the program director role at the National Science Foundation (NSF) presented itself. | have been a 


reviewer for various funding agencies before this, and it always intrigued me as to how funding decisions 
were made beyond the reviewers’ scores. This was a great chance for me to learn something new in a uni- 
versity-like environment, with the added benefits of living in the Washington DC area. 


Having been serving as a program director for a year and half now, I can say that this job is really everything that | 
thought it would be, plus a little more. | would say that there are three main things that | learned or was re- 
minded of as a result of being a program director. First and foremost, there are so many cool topics to study 
within the realm of organizational science, and there are so many different ways to study these topics too. When 
l-O psychologists think about organizational science, we think about issues related to the “I” and the “O” side, 
and we are armed with our typical research methodologies of surveys, experiments, and observations. But there 
are many other topics that can be studied, and with the technological advancement, many different ways to 
gather and analyze data. The best part about my job is to read through proposals submitted by investigators 
from different fields and to learn about how they approach the diverse research problems related to organiza- 
tional science. A second thing that | have been reminded of is the importance of knowing your audience. Differ- 
ent funding agencies have different interests and priorities. A successful proposal needs to not only be grounded 
on solid science but also be written in a way that addresses the funding agency’s needs. Finally, | appreciate my 
colleagues and fellow program directors at the NSF very much. | am surrounded by scholars from different disci- 
plines and have different perspectives on the same issue. It’s fun to have conversations and debates and learn 
about how economists or political scientists or sociologists approach the same problem. As | wrap up my rotation 
at NSF, | am excited to take these new insights back to my home department to share with my colleagues and 
students. | am also excited to start new research projects that apply some of the new skills that | learned. 


Russ Johnson: On Becoming an Associate Editor at AMR and JBP 


With tenure comes the possibility of many new adventures, one of which is serving as an associate editor. 
Unlike pretenure, when all of your time and effort really ought to be devoted to pushing your own research 
out, expectations for service and “giving back” to the academic community take on a bigger role for tenured 
faculty. Initially, the way | gave back was mostly by reviewing for journals, first as an ad hoc reviewer and 
then as a board member. However, after consistently reviewing for a journal (and providing timely and con- 
structive reviews!), you may be invited to join the associate editor team at that journal, which | was fortunate 
to experience at Journal of Business and Psychology and Academy of Management Review. Although receiv- 
ing that invitation is an honor, it can also precipitate thoughts of “What the heck am | getting myself into if | 
say ‘yes’ to this?” | thought I’d share what | see as some of the key challenges and benefits of being an associ- 
ate editor (I'll start with the challenges to end on a positive note!). 


Without a doubt, the biggest challenge is the time commitment. Serving as an associate editor at a major jour- 
nal means you could be handling anywhere from 30 to 60 (or more!) new submissions per year, plus whatever 
revise-and-resubmit papers are in your pipeline. Factoring in the time it takes to process submissions (e.g., read- 
ing manuscripts, assigning reviewers, integrating reviewer comments, writing decision letters, etc.), | would 
spend about one day per week on my associate editor duties. That’s a lot of time, especially considering you still 
have to teach, mentor students, and make progress on your own research! Another challenge is making the 
switch from a reviewer mindset to an editor mindset. As a reviewer, the mindset tends to be negative and pre- 
vention oriented (e.g., identifying a study's problems and reasons for rejecting it), whereas an associate editor’s 
mindset must be more constructive and promotion oriented (e.g., identifying a study’s strengths and feasible 
ways to redress problems). After years of judiciously searching for a study’s problems and limitations as a gradu- 
ate student in seminars and then as an ad hoc reviewer and board member, it required no small effort to over- 
come this prevention-oriented inclination and make the needed switch to a developmental mindset. 


So why succumb yourself to the time and effort demands of being an associate editor? Well, doing so 
provides plenty of benefits. For one, it gives you the opportunity to have a direct and meaningful impact 
during the review process by deciding whether and how a submission should be revised. As a reviewer, 
there were several instances where | thought rejected submissions had high potential, but | lacked the 
decision-making authority to give the authors a second chance. As an associate editor, you get to make 
that call! Another benefit is, after dealing with numerous submissions, you develop a much better sense 
of effective (and ineffective) strategies for problematizing research questions, articulating contributions, 
and conducting comprehensive analyses. As a result, my own writing and responses to reviewer com- 
ments have improved tremendously. Other benefits include broadening your knowledge when you han- 
dle manuscripts outside your primary areas of expertise and gaining further insight into the publication 
process and what goes on “behind the scenes.” In short, serving as an associate editor was a rewarding 
(albeit challenging) endeavor that involved new responsibilities and different skill sets, which provided a 
nice change of pace from my earlier pretenure experiences. 


Chris Rosen: On Becoming an Elected Officer at AOM 


| was asked to discuss what motivated me to take on leadership positions in professional organizations 
outside of my university. | have been asking myself this question quite a bit lately, as | am currently hav- 
ing a very busy week as the program chair for the HR Division of AOM (a big shout out to everyone who 
submitted their assigned reviews on time!). As anyone who has held office in SIOP will tell you, there are 
challenges associated with leadership roles in our professional organizations. These include balancing 
ongoing research projects, teaching obligations, and departmental and university service with the de- 
mands associated with external leadership roles. In my case, these demands have ranged from oversee- 
ing the development of a new website (I am a self-proclaimed luddite) to tracking down emergency re- 
viewers and developing cogent paper sessions for the conference program (that's what | worked on this 
past weekend). You read that correctly—as a tenured professor, | voluntarily spent this past weekend 
working on something that is appreciated, but not required, by my university. 


So, why have I taken on leadership roles in our professional organizations? Unlike Indiana Jones, | do not 
do this for fame and glory. Rather, | do it so that | can count myself among the numerous volunteers who 
are necessary for our professional organizations to thrive. Without members volunteering to serve on 
committees or take leadership roles, organizations such as SIOP and AOM would not be able to provide 
high quality resources and content (e.g., workshops, paper sessions, symposia). | have benefited greatly 
from having access to these resources and the volunteers that make them possible. As such, when my 
name was called, | felt that it was important for me to do my part by taking on a leadership role. In so do- 
ing, | would be able to give back to an organization that has benefitted me greatly over the years. At the 
same time, | would have the opportunity to work with others to shape the organization going forward and 
to ensure that it continues to provide excellent resources and opportunities to members. 


In addition to viewing this as a way of giving back to an organization that has contributed to my profes- 
sional success, | have also taken on leadership positions for less altruistic reasons. In particular, volun- 
teering for professional organizations has provided a terrific opportunity for broadening my social net- 
work. | have been fortunate to work with a number of fantastic people while serving on the leadership 
track of the HR Division, including David Allen, Ingrid Fulmer, Maria Kraimer, Dave Lepak, Anthony 
Nyberg, and Deidra Schleicher. Working with these folks to address the myriad challenges that we have 
encountered as a professional organization has been among the most rewarding things that | have done 
in my career. | will conclude by saying that working through issues with leadership teams in our profes- 
sional organizations is much more fun than attending a faculty meeting—trust me! 


Max. Classroom Capacity: The Dreaded Group Project 


Loren J. Naidoo 
Baruch College and the Graduate Center, CUNY 


Like most of you, I’ve spent much of February absorbed by the thrilling coverage of Olympic curling, an athletic 
spectacle unlike any other. As a good Canadian, I’ve spent many an evening screaming “Hurry! Hard!” at the TV in 
a fit of ecstatic simpatico with the vigorously sweeping titans gliding down the sheet, graceful as a cheetah scratch- 
ing its chin. The strategizing, the polite sportsmanship, the coordinated yelling—these are groups working at their 
groupiest! Coincidentally, | have just started overseeing a set of student group projects in my undergraduate re- 
search methods class, with hopes for similarly lofty levels of excitement and achievement. 


In fact, when I reflect on my academic career, | realize that almost every class | teach has a group project of some 
sort, but I’m not sure how much thought or planning went into that decision at the time. Thinking about it now, | 
can come up with a couple of reasons: 


1. Students should learn to work together because most likely they will have to do that in their work and 
personal lives, and it’s nice to make new friends. 

2. For some groups in some classes, a kind of magic happens and the group is able to create something that 
greatly exceeds expectations—the group can be greater than the sum of its parts. 


The flip side is that for some other students, group projects can be an academic nightmare like no other. In a spirit 
of well-intentioned masochism, | decided to dig up past student evaluation feedback on various group projects: 


“The group project is long and dull” 
“The (group) project is beyond torment” 


(Some say the same of televised curling—pure balderdash and calumny!) 
But somewhat to my surprise | found quite a few positive comments too: 


“The thing | appreciate most about the class is the group work” 
“The group projects help me to understand how to apply what we have learnt” 


My original goal for this column was to provide a detailed report on the issues, considerations, controversies, and 
empirical evidence concerning the use of Collaborative Learning techniques, in about 2000 words. This, of course, 
is impossible. So instead, like my Olympic heroes and heroines, we will skim lightly across a slippery surface, with 
the hopes that some of our stones (of wisdom) will end up “on the button.” 


What are group projects? 


Without getting bogged down, let’s define group projects as a particular case of collaborative learning in which 
evaluated classroom activities involve three or more students actively communicating and cooperating with each 
other towards a common goal, under the guidance of the instructor. OK? OK. While we’re defining things, curling 
can be defined as shuffleboard + chess on ice on steroids! 


Why use group projects? 


There are circumstances in which group work is beneficial for practical or structural reasons. It can allow students to 
work on projects which would be too much work to complete individually, such as conducting an empirical study in a 
research methods class, or a consulting project in an I-O psych or management class. Group projects may also reduce 
the grading burden of the instructor and/or TA, though if they are designed well, probably not by much. 


Various theoretical reasons for the value of collaborative learning also have been proposed, including the notion 
that knowledge encoding is superior when occurring in a rich social context (Olmstead, 1974), collaborative work 
promotes the development of positive interpersonal relationships with fellow students (including multicultural re- 
lations; Aronson, 1971), self-esteem, social support, and positive attitudes towards school (Johnson et al., 2007). 


In addition, the perceived value of teamwork skills to the workplace is evident in the attention they receive in the 
training/teams literature (e.g., Chen et al., 2004; Stoner et al., 2015) and in I-O and business classrooms. This value 
also is reflected in teamwork skills appearing on the list of AACSB learning standards. In sum, teamwork is valued 
by employers, and thus those who are concerned with preparing students for the work world tend to see the im- 
portance of developing student teamwork skills. 


Are group projects effective? 


There is a large literature in education, including several prominent meta-analyses (e.g., Johnson et al., 1991, Pae et al., 
2015; Sung et al., 2017) which consistently supports the learning effectiveness of collaborative (versus competitive or 
individualistic) learning at various educational levels and across diverse contexts (see Johnson et al., 2007 for a review). 
As an example of evidence more specific to psychology, Tomcho and Foels (2012) conducted an underpowered meta- 
analysis of studies on group learning activities in psychology classes published in the Teaching of Psychology Journal be- 
tween 1974 and 2011. They found a large positive effect for collaborative learning activities on outcomes including 
graded performance, nongraded knowledge, self-reported attitude/belief change, and behavior change. 


What are the ideal conditions for group work? 


Wait, wait, wait—what about social loafing? Doesn’t that make group projects risky? Yes, social loafing is a real 
concern, but here are some guidelines (e.g., read Meyers, 1997) for how to avoid it, in brief: (a) The task must be 
divisible so that different students can take on different aspects; (b) the task must be additive such that contribu- 
tions from every student are needed; and (c) individual performance must be identifiable, evaluated, and incentiv- 
ized. To put it in more familiar curling terms: when you have the hammer in the tenth end and you need a simple 
hit and stay for the win but you throw a hog so the other team steals two, it’s pretty clear that you’re the hoser, 
eh? 


For example, in addition to submitting a single paper for their group projects, | require students to submit evalua- 
tions of the contributions to the project of all of the group members including themselves. When students rate 
others (or themselves) poorly, | ask them to carefully describe the specific incidents or behaviors that led to the 
low ratings. | carefully examine these ratings and comments, and where the pattern of evidence indicates that one 
or more students were loafing, | adjust individual grades accordingly. This is a lot of work, especially when there 
are big disparities in the ratings. | have found it a worthwhile investment to help groups at the beginning to devise 
reasonable ways to divide up the work and assign specific tasks and deadlines. It also helps to make students well 
aware of the fact that their individual contributions to the group project matter, and that failing to do one’s part 
will result in a lower grade. When | take the time to do this, the majority of groups seem to function well, which 
means they probably learn more, and there are far fewer ratings disparities to deal with. There are many other 
ways to achieve these ends, and Meyers (1997) provides a nice description of many of them. 


There are many other questions to ask about how to design group projects, including: what are the optimal group 
size, project duration, task complexity, group and/or individual reward/accountability structure, and peer and/or 
instructor assessment methods? Here the literature yields disparate findings and many effects look to be context- 
specific. For example, Sung et al. (2017) found generally larger positive effects for collaborative learning on learn- 
ing outcomes with groups of 4 and more than 4 compared to groups of 3 or 2. In contrast, Tomcho and Foels 
(2012) found no moderating effect for group size. Similarly, Sung et al. found that effect sizes were larger for pro- 
ject durations of roughly between 1 week and 6 months than for those less than 1 week or more than 6 months. In 
contrast, Tomcho and Foels found larger effect sizes for projects one to three class sessions long compared to half 
a semester or greater. Primary studies in different contexts also tend to produce different effects for these varia- 
bles. There does not appear to be a simple story—perhaps we will revisit this issue in some future column. 


Can technology help? 


Yes! Only fairly recently did | get around to removing a standard line in my syllabi asking students to turn off their 
pagers in class. Technology advances! Mobile phones provide ease of access to a variety of helpful collaboration 
tools, especially in the context of hybrid/blended and fully online classes. Today | met with two groups of students 
to discuss their first group project in an asynchronous fully online research methods class. Most of these students 
are taking research methods fully online precisely because it is very difficult for them to come to campus for class 
or other meetings. So we met using Zoom, a platform similar to Skype which allows group audio/video conferenc- 
ing, screen sharing, whiteboarding, hand raising, and so on. There are many other tools. 


There are also people who conduct empirical research on the effectiveness of Mobile-Computer-Supported Collab- 
orative Learning (mCSCL). For example, Sung et al. (2017) meta-analyzed 16 years of empirical research on the 
topic. In case you are wondering how long smart phones have existed (like | did when I read that paper—| googled 
it so you don’t have to!), blackberry phones with e-mail were out by 2003 and the iPhone was released in 2007; 
their definition of mobile includes laptop computers and PDAs. They examined the outcomes of learning achieve- 
ment, learning attitude, and peer interaction. They found large positive effects for mCSCL over noncollaborative 
learning and computer-less collaborative learning, and a smaller effect over nonmobile CSCL. The benefits for 
mCSCL were somewhat greater with larger group sizes suggesting mobile technologies mitigate some of the pro- 
cess losses due to a lack of social cohesion, lack of coordination or increased diffusion of responsibility associated 
with larger group sizes. 


Nowadays most students will use mobile technology to help with group projects whether you intend them to or 
not, to their benefit—all the more reason to assign such projects! You know what else technology lets you do? Re- 
play entire Olympic curling matches on your phone... 


What are some ideas for innovative and effective group projects in my I-O psychology/management class? 


Muir and van der Linden (2009) describe an incredible group project they developed that involved groups of 3 to5 
introductory psychology undergrads preparing and delivering a short lecture on psychology to local elementary 
school students. The task required students to deeply understand the target material and how to present it to a 
developmentally younger audience. This approach is particularly powerful as it combines collaborative, experien- 
tial, and service learning. 


Another relatively intensive approach is Aronson’s (1971) jigsaw classroom. A lesson is divided into a certain num- 
ber of parts—let’s say five as an example. The class is divided into groups of 5 students, each tasked with learning 
one of the five parts. All students assigned the first part get together to learn it, same with those assigned the sec- 
ond part, and so on. Then, each student returns to their jigsaw group, and teaches their part of the lesson to the 
four remaining group members. Here the task is divisible, interdependence is baked into the structure, and individ- 
uals are accountable for their own learning and that of their group mates. 


If these sound like too much of a commitment, about 12 years ago | created a group project for my large social psy- 
chology classes which involved small groups of students creating a short and entertaining educational video about 
a social psychology theory or study and posting it on a then obscure website called YouTube. The idea is similar to 
Muir and van der Linden’s (2009) (i.e., students as teachers) though | don’t doubt the superiority of their approach. 
In full disclosure, this was probably the project that was deemed “beyond torment” by one articulate student. 
However, some groups produced incredibly creative, clever, funny, moving and insightful videos that I’m certain 
had a greater impact on their long term retention of that material than anything else in the class. 


Please tell me about your experiences with group projects, good and bad: Loren.Naidoo@baruch.cuny.edu. And 
keep your broom on the sheet. 
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SIOP in Washington: Advocating for I-O in Federal Public Policy 


Since July 2013, SIOP and Lewis-Burke Associates 
LLC have collaborated to make I-O science and 
research accessible to federal and congressional 
policy makers. SIOP has embedded a foundational 
government relations infrastructure within the 
organization, enabling SIOP to develop an 


authoritative voice as a stakeholder in science policy 
in Washington, D.C. and to promote SIOP as a vital 


Jill Bradley-Geist, University of 
resource for evidence-based decision making. 


Colorado Colorado Springs, and Bill 


Ruch, Lewis-Burke Associates LLC 


Appropriations Update: Congress Reaches Budget Deal and Trump Administration Lays Out Spending 
Priorities 

On February 12, President Trump released his second budget proposal to Congress. The fiscal year (FY) 
2019 budget request reflects the political priorities of the White House, but it is ultimately up to 
Congress to decide which proposals to embrace, modify, or reject as part of the annual appropriations 
process. 


Shortly before the budget was released, Congress and the White House agreed on new funding caps for 
both FY 2018 and FY 2019 in the Bipartisan Budget Act of 2018, which has paved the way for completing 
FY 2018 appropriations and establishing a FY 2019 budget framework. Although Congress is hoping to 
wrap up negotiations in March on FY 2018 funding levels for individual federal agencies, the president’s 
budget request seeks to set an agenda for FY 2019, accompanied by a $200 billion infrastructure 
package. Ultimately, Congress will decide FY 2019 funding levels, but the nonbinding budget request 
provides a window into activities and planning underway within the agencies and what programs will 
receive the most attention and focus in the year ahead. 


With respect to funding levels proposed in the president’s request, the Department of Defense (DOD) 
would be the clear winner, including a 19% requested increase for the Defense Advanced Research 
Projects Agency (DARPA). With the higher spending caps just approved, this budget request would 
provide mostly flat funding (the same level as FY 2017) for many of the largest civilian research agencies, 
such as National Institutes of Health (NIH), the National Science Foundation (NSF), and the Department 
of Energy’s (DOE) Office of Science. This is in contrast to Congress, which is expected to use the higher 
caps to increase many of these agencies’ budgets in the final FY 2018 appropriations bill. Not all 
programs fare as well in the Trump budget request. In a repeat of last year’s recommendation, the 
request would eliminate several agencies and programs, such as the National Endowment for the 
Humanities (NEH) and the National Endowment for the Arts (NEA). These recommended eliminations 
are expected to again be rejected by Congress. 


The budget request also seeks to impact education and workforce programs, including more than 
doubling the Department of Labor’s (DOL) apprenticeship program. Similar to last year, the Department 
of Education (ED) request would decrease funding that supports college access and affordability, in part 


to offset proposed increases for programs that enable public and private school choice. Most notably, 
the request would eliminate the Federal Supplemental Educational Opportunity Grant program, GEAR 
UP, and Title VI International Education programs while significantly reducing funding for Federal Work- 
Study. 


Overall, President Trump’s FY 2019 budget request would build on the previous year’s emphasis on 
defense and national security at the expense of nondefense federal agencies and programs. The budget 
request also proposes further agency streamlining, reorganizations, and staff reductions. These changes 
reflect a multimonth process by the White House Office of Management and Budget (OMB) to shrink 
the size of the federal government and trim the number of personnel and offices required to conduct 
oversight or manage programs. Some of these staff reductions have already begun to occur, through 
accelerated “buy outs” of personnel, restructuring of offices, or staff attrition. 


Lewis-Burke Associates LLC and SIOP will continue to monitor the situation and seek opportunities to 
engage on behalf of crucial research programs as deliberations continue. 


SIOP Government Relations Introduces the Veterans’ Transition Initiative to Congressional Staff 


In December, Adam Kabins joined Lewis-Burke Associates LLC (Lewis-Burke) for a series of meetings 
with representatives from Capitol Hill to highlight the impacts and applications of l-O evidence-based 
research to assist veterans as they transition to the workforce; urge stakeholders to apply I-O research 
to related policies and programs; and position and promote SIOP as a collaborator and resource for 
these offices going forward. 


The first wave of outreach meetings facilitated by Lewis-Burke were with bipartisan, bicameral 
congressional offices that have been heavily involved in the ongoing conversation over veterans’ 
transition to the civilian workforce, including the House and Senate Veterans’ Affairs Committees, 
respectively. The meetings were a complete success, as staff from each office expressed interest in 
learning more about the information provided in the Veterans’ Transition Advocacy Statement, a 
guidance document developed by Lewis-Burke in conjunction with SIOP that outlines the critical 
importance of I-O research-backed solutions to enhance veteran employment experiences. Further, 
several committee staff pledged to consider engaging SIOP as a consultative resource in future 
discussions on veterans’ workforce transition legislation that committees have begun to develop. 


Through these meetings, the group learned that both House and Senate committees have been working 
with the Department of Defense (DOD), the Department of Veterans Affairs (VA), as well as other 
private organizations to focus on both veterans’ transition to and retention in civilian jobs. Several 
committee staff believed that I-O research could be helpful in establishing metrics to measure a 
veteran’s suitability for a civilian job and the development of strategies for translating military 
competencies a veteran has learned to civilian jobs. Lewis-Burke and SIOP will continue to interact with 
these offices and maintain the relationships built through the meetings. 


This outreach was part of an ongoing collaboration between Lewis-Burke and the Government Relations 
Advocacy Team (GREAT) at SIOP. Lewis-Burke leads government relations outreach and seeks 
opportunities to profile I-O findings to federal stakeholders interested in veterans’ transition. These 
efforts are in close collaboration with a team of SIOP experts led by Adam Kabins. The team will 
continue to coordinate SIOP efforts and I-O research and practice findings relevant to the transition of 


veterans to the workplace and provide timely feedback and expertise to Lewis-Burke on pressing federal 
issues as they arise. 


Message from Government Relations Advocacy Team Chair, Jill Bradley-Geist: How does your work 
relate to public policy? 


On February 16, | had the opportunity to visit my State Capitol in Denver and participate in meetings 
where we heard from representatives and economists on pressing issues within the state (and also local 
and federal) government. Speakers included State Senator Owen Hill (R), State Representative Edie 
Hooton (D), State Representative Bob Rankin (R), and Natalie Mullis, Fiscal Director, Colorado Legislative 
Council. 


A common theme of the remarks was the importance of the public’s involvement in the legislative 
process. In particular, the Representatives welcomed input from academic and practitioner subject 
matter experts on issues related to key policy concerns. Examples of topical issues mentioned throughout 
the meetings included DACA, cybersecurity, workforce readiness as related to training and education, 
technology and automation in the workplace, population demographics, and the aging 
population/workforce. Several of these topics are rooted in I-O psychology and demonstrate the need for 
our science to be recognized by policymakers and considered in the development of policies at all levels 
of government, which are enduring goals for the Government Relations Advocacy Team (GREAT) and 
SIOP leadership. 


We would love to know how your own work relates to government and policy issues, and if you have 
been involved in government advocacy at the local, state, or federal level! To share your thoughts and 
experiences, please contact Government Relations Advocacy Team Chair, Jill Bradley-Geist at 
jbradle3@uccs.edu. Also, please join us this April at SIOP 2018 Annual Conference in Chicago for a special 
session on SIOP Government Advocacy and linking your own work to advocacy and public policy issues 
(Friday, April 20, 3:00 — 3:50 PM — Wrigleyville: Executive Board Session: Link Your I-O Work to Federal 
Policy & Funding Opportunities). 


Neuroscience, Outgroups, and Bad Behavior 


Bill Becker 
Virginia Tech 


M.K. Ward 
University of Western Australia 


Xiaoyuan (Susan) Zhu 
University of Connecticut 


As the recent “Me Too” movement dramatically demonstrates, the modern workplace remains rife with 
reprehensible behavior that extends from negative politics, to abusive supervision, to sexual harass- 
ment. A recent Forbes article suggests that as many as 75% of employees are affected by workplace bul- 
lying. The persistence and extent of bad behavior suggests that it cannot be attributed to a few bad ac- 
tors but rather that all of us are susceptible to our inner demons and there is reason to believe that so- 
cial categorization may play a role in our Jekyll and Hyde transformations. 


l-O psychology and social psychology has long recognized and explored the role of social categorization 
and in/out-groups in the workplace. The consequences can be ugly, including forms of aggression and 
stereotyping (Hogg & Terry, 2000; Nelson, 2009). Despite previous investigations, we still don’t have a 
full understanding of how these categorizations are made in work situations. Recent findings in neuro- 
science give new insights into this question. Spoiler alert: There can be drastic effects of these categori- 
zations on brain processing. In this article, we will review these findings and suggest some ways that 
they might be incorporated into organizational research and practice. 


Social Categorizations Are Often Nonconscious 


One intriguing finding across a number of studies is that social categorization evaluations are rapidly ini- 
tiated at an implicit level. A large number of these studies have focused on racial outgroups and suggest 
that is an implicit tendency to categorize members of different races into outgroups (Cunningham et al., 
2004; Phelps et al., 2000). These studies have consistently shown differences in emotional processing for 
unfamiliar individuals from different racial groups. Preliminary findings suggest that implicit categoriza- 
tions also extend to gender and other social differentiators (Chiao et al., 2008). 


However, one study found that minority members also showed negative implicit response to pictures of 
unfamiliar male minority individuals (Lieberman, Hariri, Jarcho, Eisenberger, & Bookheimer, 2005). This 
has given rise to the perspective that although initial implicit categorization processes are influenced by 
race and other observable differences, subsequent processing and out-group determinations are more 
heavily influenced by threat perceptions relative to the self and other in-group members (Chang, Krosch, 
& Cikara, 2016; Checkroud, Everett, Bridge, & Hewstone, 2014). Still, it seems that social categorizations 
and out-group determinations occur primarily through emotional processing rather than explicit cogni- 
tive processing. The threat perspective of outgroups extends social categorization beyond discrimination 
to a broader range of organizational applications. 


Outgroup Categorizations Are More Impactful When the Stakes Are High 


In addition, the level of perceived threat also has a profound effect on the behavioral tendencies toward 
out-group members. Under conditions of low threat and plentiful resources, people tend to view out- 
group members with indifference (Cuddy, Fiske, & Glick, 2007). However, when threat is perceived to be 
greater and resources are scarce, people respond to out-group members with active hostility (Chang et 
al., 2016). This suggests that out-group effects are situationally dynamic in terms of their makeup and 
effects. 


Outgroup Categorizations Shutdown or Reverse Social Processing in the Brain 


Two related research streams help inform these effects by demonstrating the profound effect of out- 
group categorization on brain processing. The first relates to oxytocin, a neurotransmitter that influences 
social behavior. It has frequently been associated with increased trust, cooperation, and prosocial behavior 
(Bartz, Zaki, Bolger, & Ochsner, 2011). However, more recently these effects have been shown to hold pri- 
marily toward in-group members. In contrast, oxytocin has been shown to decrease trust and cooperation 
toward out-groups (De Dreu, 2012). Consistent with the threat perspective, oxytocin also increased defen- 
sive aggression toward outgroups (De Dreu et al., 2011). These findings show that social categorization can 
reverse the effects of oxytocin, a neurotransmitter beloved by many as the love hormone. 


A related stream of research shows out-group categorization also influence the way empathy is processed in 
the brain. The automatic experience of empathy towards others is an important aspect of emotional intelli- 
gence. Numerous studies indicate that empathy is much stronger for ingroups than for outgroups (Chang et 
al., 2016). Once again, consistent with the threat perspective, when intergroup threat is high people often 
experience counterempathy (schadenfreude) in response to outgroups. Once again, the neuroscience evi- 
dence supplements these findings by showing that brain processing of empathy is enhanced for ingroups and 
essentially turned off or even reversed for threatening out-group members by dehumanizing them (Harris & 
Fiske, 2006; Xu, Zuo, Wang, & Han, 2009). A key take away here is that both research streams highlight the 
power of context and framing to flip the relationships between neural activity and behavior, effectively turn- 
ing a normal individual into a psychopath toward outgroup members. 


Implications for Research and Practice 


Although the basic research has tended to focus on racial outgroups, the findings can be generalized to 
the workplace in a number of promising ways. First, it seems that organizational power and politics can 
give rise to perceptions of competitive or antagonistic relationships between coworkers and between 
leaders and subordinates. These perceptions may give rise to nonconscious categorizations of out-group 
status. Second, a better understanding of the dramatic effects this can have could explain the promi- 
nence of abusive supervision, sexual harassment, and workplace bullying that continues to plague or- 
ganizations. It may also inform research on organizational justice and ethics by emphasizing the im- 
portance of emotions and the impact of in/outgroups on emotional processing and subsequent unethi- 
cal behavior. Third, this stream of research can increase our understanding of the impact of social cate- 
gorization on employee selection and promotion. Applicants and employees who are considered to be 
in-group members may be more likely to be selected for a job and promoted to a higher position (Da- 
vison & Burke, 2000). 


From a practical perspective, these insights could inform organizational education and training to in- 
crease awareness of the impact of social categorization and provide individuals at all levels of organiza- 
tions with tools for expanding their ingroups and shrinking outgroups while also avoiding the bias traps 


of in-group love and out-group hate. This research is particularly important for human resources profes- 
sionals who work actively to minimize discrimination in the workplace. Finally, work design may be able 
to reduce such biases by fostering identity development and expansion from narrow to collective identi- 
ties (Parker, 2014). Overall, we see that both neuroscience and work context are essential tools in our 
quest to reach a full understanding of the processes that create and sustain social categorizations. 


References 


Bartz, J. A., Zaki, J., Bolger, N., & Ochsner, K. N. (2011). Social effects of oxytocin in humans: Context and person 
matter. Trends in Cognitive Sciences, 15(7), 301-309. 

Chang, L. W., Krosch, A. R., & Cikara, M. (2016). Effects of intergroup threat on mind, brain, and behavior. Current 
Opinion in Psychology, 11, 69-73. 

Chekroud, A. M., Everett, J. A., Bridge, H., & Hewstone, M. (2014). A review of neuroimaging studies of race-related 
prejudice: does amygdala response reflect threat? Frontiers in Human Neuroscience, 8, 179-190. 

Chiao, J. Y., Adams Jr, R. B., Tse, P. U., Lowenthal, W. T., Richeson, J. A., & Ambady, N. (2008). Knowing who's boss: 
fMRI and ERP investigations of social dominance perception. Group Processes & Intergroup Relations, 11(2), 
201-214. 

Cuddy, A. J., Fiske, S. T., & Glick, P. (2007). The BIAS map: behaviors from intergroup affect and stereotypes. Jour- 
nal of Personality and Social Psychology, 92(4), 631. 

Cunningham, W.A., Johnson, M.K., Raye, C.L., Gatenby, J.C., Gore, J.C., & Banaji, M.R. (2004). Separable neural 
components in the processing of Black and White faces. Psychological Science, 15, 806-813. 

Davison, H. K., & Burke, M. J. (2000). Sex discrimination in simulated employment contexts: A meta-analytic inves- 
tigation. Journal of Vocational Behavior, 56(2), 225-248. 

De Dreu, C. K. (2012). Oxytocin modulates cooperation within and competition between groups: an integrative re- 
view and research agenda. Hormones and Behavior, 61(3), 419-428. 

De Dreu, C. K., Greer, L. L., Van Kleef, G. A., Shalvi, S., & Handgraaf, M. J. (2011). Oxytocin promotes human ethno- 
centrism. Proceedings of the National Academy of Sciences, 108(4), 1262-1266. 

Harris, L. T., & Fiske, S. T. (2006). Dehumanizing the lowest of the low: Neuroimaging responses to extreme out- 
groups. Psychological Science, 17(10), 847-853. 

Hogg, M. A., & Terry, D. I. (2000). Social identity and self-categorization processes in organizational contexts. Acad- 
emy of Management Review, 25(1), 121-140. 

Lieberman, M.D., Hariri, A., Jarcho, J.M., Eisenberger, N.I., & Bookheimer, S.Y. (2005). An fMRI investigation of 
race-related amygdala activity in African-American and Caucasian-American individuals. Nature Neuroscience, 
8, 720-722. 

Nelson, T. D. (Ed.). (2009). Handbook of prejudice, stereotyping, and discrimination. New York: Psychology Press. 

Parker, S. K. (2014). Beyond motivation: Job and work design for development, health, ambidexterity, and 
more. Annual Review of Psychology, 65, 661-691. 

Phelps, E. A., O'Connor, K. J., Cunningham, W. A., Funayama, E. S., Gatenby, J. C., Gore, J. C., & Banaji, M. R. (2000). 
Performance on indirect measures of race evaluation predicts amygdala activation. Journal of Cognitive Neuro- 
science, 12(5), 729-738. 

Xu, X., Zuo, X., Wang, X., & Han, S. (2009). Do you feel my pain? Racial group membership modulates empathic 
neural responses. Journal of Neuroscience, 29(26), 8525-8529. 


Awards Spotlight: The Path to Fellow—Dr. Leslie Joyce 


In this installment of the T/P Awards Spotlight, we continue our focus on the SIOP Fellow process. In the 
first article of our series, we interviewed the current chair of the SIOP Fellow committee Dr. Kenneth P. 
De Meuse to learn more about the process and his recommendations. The second article saw an inter- 
view with Dr. Cheryl Paullin’s path to becoming a Fellow. In this third installment, we continue our Path- 
to-Fellows interviews with Dr. Leslie Joyce. Receiving her doctorate from North Carolina State Univer- 
sity, Dr. Joyce has spent most of her career in the talent management space specializing in employee 
and organizational development and effectiveness. Dr. Joyce’s employee development emphasis is not 
surprising giving her half namesake role in the likewise-development-focused Leslie W. Joyce and Paul 
W. Thayer Graduate Fellowship in l-O Psychology. More recently, Dr. Joyce has transitioned into leading 
global human resource functions, particularly as the chief people officer for a global manufacturing com- 


pany. 
Dr. Leslie Joyce’s Path to SIOP Fellow 


Becoming a SIOP Fellow requires an unusual and outstanding career as applied to 
one or more of five specific areas: research, practice, teaching/education, service, 
and administration. A candidate’s qualifications are summarized collectively in a six- 
part professional dossier including a self-statement, curriculum vitae, nomination 
letter, endorser list, and endorser letters. It is not uncommon, notes De Meuse, for 
an ultimately successful application to be built over the span of years with the help 
of one’s colleagues (Howardson & Munson, 2017). Indeed, Dr. Joyce confirmed as much (noted below by 
A) when we asked (denoted below by Q) about her Fellows experience. 


Q: If you wouldn’t mind, please describe the “work” you did that you think played a key role in 
becoming a fellow. As follow ups, what were your innovations/unique contributions to the field? 
How did you “significantly” impact the field in line with the qualifications for becoming a Fellow? 


A: | think the key for me was to stay firmly engaged on both sides of the scientist — practitioner con- 
tinuum. | was committed to being an I-O psychologist and a leader in a corporate environment. To 
do that I had to stay current in I-O practices and be well versed in the nature of applied settings. 
That requires technical capability as well as intellectual and professional flexibility. | focused on pub- 
lishing and speaking and ensuring the implementation of I-O best practice. | was committed to mov- 
ing the practice forward (contributing to advancing the field) and in sharing those successes with 
others so they could add their efforts to advancing the field. | did a lot of work in the employee sur- 
vey area driving new levels of effectiveness in action planning, made contributions to the design of 
assessment centers, created practices for change management that raised the bar. 


In keeping with the Fellow requirements, Dr. Joyce’s career-long work has been truly unusual and out- 
standing. As De Meuse notes, one characteristic of a Fellow-worthy career is making one’s work accessi- 
ble and available to the I-O psychology field writ-large beyond one’s own immediate work setting. Com- 
municating as such is most effectively done when describing unambiguous contributions, preferably, us- 
ing specific metrics quantifying the candidate’s extended impact. An unusual and outstanding career, in 
other words, requires not only individual success but also elevating the work of one’s peers to support 
overall advancement of the I-O psychology field. So it has been with Dr. Joyce’s career by publishing, 
speaking, and implementation work focused on helping others “add their efforts to advancing the field.” 


It is important to note that identifying such efforts solely from one’s own perspective can be challeng- 
ing. As De Meuse has previously noted, the importance of feedback from one’s peers throughout the 
Fellows application process should not be understated. Indeed, as noted below, Dr. Joyce echoed a simi- 
lar recommendation when recounting her own path to SIOP Fellow, particularly as related to writing 
one’s own self-statement. 


Q: What did you include in your self-statement? 


A: This was far harder than | anticipated. It is so hard to think of what you do every day as special, 
unique or noteworthy. It’s just what you did. It really helped me to get other people’s perspective 
on my contributions so | could see them objectively for what they contributed to the field. Once | 
distanced myself from myself—I could see more clearly. The key elements | included in my state- 
ment highlighted my work in leadership development, employee engagement and talent manage- 
ment as well as my contributions through publishing and speaking. A key element was illustrating 
how I had broadened the audience for I-O by bringing our perspective to other fields. 


In a similar vein, De Meuse notes that the most effective self-statements are those adopting a more 
neutral and objective perspective about one’s contributions using, for instance, specific metrics (e.g., 
published new assessment center structure adopted by x number of peers in other organizations/indus- 
tries). Note, however, that objectively stating one’s contributions need not be overly modest. In fact, Dr. 
De Meuse notes that the self-statement is precisely the appropriate time to promote one’s career ac- 
complishments. Remember, the criteria for becoming a SIOP Fellow do not require a “good” or “success- 
ful” career but rather an unusual and outstanding career, the communication of which might be chal- 
lenging without highlighting or promoting career accomplishments in kind. 


As both Drs. Joyce and De Meuse note, the Fellowship onus is not to be assumed alone. Indeed, when 
asked to offer recommendations for others pursuing the Fellow path, Dr. Joyce emphasized the im- 
portance of one’s colleagues and remaining active in the broader |-O community. 


Q: What advice would you give to those interested in becoming a fellow? 


A: There are so many terrific role models to choose from. Get to know them—what they did and 
how they did it. Keep a well-balanced network (both practitioners and academics) and value all 
the members of that network. Give back to the Society by publishing, speaking and mentoring. 


Q: What do you think is the “secret sauce” to becoming a Fellow as practitioner? 


A: | think the secret sauce is to demonstrate commitment to the science and to build the bridges 
between the science and its application that move the needle on excellence in corporate prac- 
tices. | also think it’s important to be involved in the Society—to publish, to speak, and to men- 
tor within the profession. 


Perhaps the most important colleagues in the Fellow process are those of the nominator and the endors- 
ers. As Dr. Joyce notes below and as previously noted by Dr. De Meuse, the relationships between nomina- 
tor, endorsers, and candidate should not be tangential; successful Fellow applications include endorse- 
ment letters from several colleagues intimately familiar with the candidate’s career work. Further, identify- 
ing colleagues qualified as such is less a function of the endorsers’ overall notoriety in the field and more a 


function of their knowledge with the candidate’s work. Such qualifications are best adjudicated with help 
from the candidate’s nominator, which, as noted below, was precisely Dr. Joyce’s experience. 


Q: Can you describe the nomination process? How did you get nominated? Did you find someone 
to nominate you, or did someone else nominate you? How did you find people to endorse you 
or did your nominator do that? 


A: My nomination came straight out of the blue, and | was so honored. It was something | did not 
think | would achieve in my career as it is the pinnacle of professional and academic recognition 
for us. A colleague that | have known and worked with for many years nominated me. He and | 
worked together on who to contact for endorsements and then he took it from there. It was 
quite nerve wracking to go through the process, but I’m very honored. 


Summary and Conclusion 


As noted by De Meuse and confirmed by our interview with Dr. Joyce, the process of becoming a Fellow 
is necessarily collaborative requiring colleagues intimately familiar with one’s work. To be sure, one’s 
colleagues acquire such knowledge not overnight, but over years. The process of becoming a SIOP Fel- 
low, in other words, really begins decades prior to compiling one’s application. As we noted in our first 
Fellows process article, “the SIOP Fellowship is an unusual and outstanding recognition covering, at 
least, a decade of one’s work and life; communicating achievements as such, not surprisingly, might take 
more than a few weeks” (Howardson & Munson, 2017). 
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Learning About Learning: The Mythical Land of I-O 
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In our last column, we talked about some of the misconceptions that I-Os may face when interacting 
with L&D professionals. As we’ve said repeatedly in this column, both sides have a lot to learn from each 
other. On the flip side of the coin, there are some I-Os who hold beliefs about workplace learning that 
don't necessarily jibe with what’s going on in the practitioner space. In this column, we’d like to talk 


1“ 


about several “myths” about training that some I-Os may have bought into over time. 


L&D Is All About Assessment and Evaluation 


This is far from the truth. Although we I-Os tend to focus heavily on using data to inform decisions, 
that’s not the primary focus of the bulk of L&D activities. There are more pieces of the puzzle to contend 
with than the narrow avenue of evaluation. Training organizations must be capable of enacting at least 
eight critical processes, including content development, training delivery, technology integration, admin- 
istrative services, portfolio management, strategic alignment, and both diagnostics and reporting and 
analysis, which encompass needs assessment and evaluation (Harward and Taylor, 2014). At a minimum, 
learning and development organizations must engage in administration to assign and track training and 
delivery before they even begin to focus on evaluation, and realistically, most of these processes must 
occur and be executed well before evaluation can come into play. In order to ensure the training is ef- 
fective, good content development and strategic alignment between processes and business goals are 
critical. In large organizations, managing learning technologies and the entire training course portfolio 
can require the attention of entire workgroups. In fact, a recent work analysis of training managers re- 
vealed that managing technology was the second most frequently performed job responsibility in the 


role (Training Industry, Inc. 2017). 


A similar myth encompasses the idea that all training programs should be evaluated using each of Kirk- 
patrick’s (1959) four levels of evaluation. The fact is, training organizations typically lack the resources 
(e.g., funding, time, personnel) needed to evaluate every program at more than one level, let alone all 
four. Perhaps this is why so many programs default to “smiley sheets” (i.e., Kirkpatrick’s level 1 reac- 
tions). The reality is (and dare we utter this sacrilege), it's probably not necessary to evaluate all pro- 
grams. Before you send us off to be reeducated on the importance of data-based decision making, hear 
us out. If a training organization is operating with a limited budget, they may not have the resources to 
offer all of the critical programs needed and evaluate them each time. If faced with a dilemma of 
whether to offer multiple learning programs designed to meet a critical need or offer only one but eval- 
uate it well, how should an L&D professional proceed? 


The answer, as with all things we encounter, is “it depends.” We think the focus in our field on evalua- 
tion and assessment is a natural and important extension of our strengths. But, we want to challenge 


our readers to consider providing more guidance on when to evaluate and at what level. Are there in- 
stances where level 1 reactions can answer the most critical evaluative questions? When is it worth the 
investment to capture data at multiple levels? Finally, consider how the data are used and offer guid- 
ance that extends beyond how to measure. For example, Kurt Kraiger and Eric Surface are adding to this 
conversation with their work investigating how to communicate value to stakeholders once evaluative 
data are collected (2017, 2018). 


L&D Professionals Have Access to Data 

Similar to the idea that L&D is all about evaluation, this myth gets at the assumption that L&D profes- 
sionals have various kinds of data at their disposal and just aren’t using it. Whether it’s data from work 
analysis activities, employee performance, attrition data, satisfaction surveys, and so on, one would be 
hard pressed to find an L&D professional able to make decisions with the benefit of all of the infor- 
mation a company has on hand. 


For example, as we’ve drilled into our graduate students time and time again, work analyses are the 
foundation of most HR systems; how often do you think those data are shared with L&D teams when 
collected? The answer, in our experience, is not often enough. Further, we know that L&D solutions can 
impact more than just performance (e.g., turnover and absenteeism, work related injuries, employee 
engagement, team cohesiveness, etc.), but are training personnel able to access that data without gath- 
ering it themselves? We hear time and time again from those practicing in the L&D field that they strug- 
gle to identify sources of such data and when they do, they face challenges convincing others to share. 


So, what’s with the data hoarding? We think there are a number of concerns driving this, including the need 
to protect employee privacy, and the possibility that data may be misused, misinterpreted, or stray from the 
stated purpose for which data were originally collected. There are also hurdles related to the labor involved 
in gathering, merging, and cleaning such data, a task that typically falls to the group holding the data rather 
than the requesting L&D team. Given these complex issues, what can we as l-Os do to help? 


We’re in a unique position to advocate for data sharing and to facilitate training professionals’ requests, 
because we understand that data-driven decisions are far more likely to result in positive organizational 
outcomes. By working closely with L&D teams, we can help them identify and better understand the kinds 
of data they may need, the formatting in which they will likely want to receive it, and the processes they’d 
need to undergo to utilize it. Those of us working in a talent analytics functions can also partner with L&D 
to provide those kinds of insights and advocate for greater use of data in L&D processes. 


Training and Education Are the Same Thing 


What’s the difference between “education” and “training”? Although your answer may go in many di- 
rections, we feel the importance lies with the overarching objective of learning. Loosely defined, educa- 
tion is concerned with teaching the theory and background of a concept; the goal isn’t necessarily for a 
learner to gain strategies for practical application, but to be exposed to information that increases their 
understanding. Training, on the other hand, seeks to impart specific skills or job-relevant knowledge to a 


learner, with the aim that the new knowledge or skills acquired in training can be applied back to a 
learner’s job. The former is typically wider in scope than the latter, but both are important to building 
productive competencies. 


If you’re thinking, “huh?” at the preceding paragraph, let’s draw out this distinction another way using a 
stats example. (Because we all love those, right?) Let’s say you’re very well versed in how a correlation 
works: what the formula is, the assumptions, matters of statistical power, how to interpret the correla- 
tion coefficient, and so on. You’re well educated about correlations but might be poorly trained in how 
to apply this knowledge to a real-world problem. On the other hand, you might be quite adept at run- 
ning a correlation in any software package, generating a scatterplot, working with outliers in the data, 
and so on, but when explaining the results to company stakeholders run afoul of the golden rule of “cor- 
relation does not imply causation.” You’re well trained in how to use a correlation, but poorly educated 
on their background. 


Why bother to draw this line in the sand? Some people may think of training as “learning things directly 
related to work” and education as “learning things that may or may not have to do with work,” with lit- 
tle else to distinguish the two. Training, obviously, is much more consequential to an individual em- 
ployee than education. Why is that? Here’s a partial list of the impacts that training can have: 


e Training outcomes can be used to inform promotions, layoffs, or other job actions; 
Training can be a critical piece of meeting regulatory and compliance requirements; 
Training upskills employees on using new pieces of software, hardware, or familiarizing them 
with new processes; 

e Training is the best vehicle for onboarding new employees to the culture and policies of an or- 
ganization; 

e Training can be used to provide knowledge and skills at the point of need (sometimes called 
“just in time” training) when a job demands it; 

e In many organizations, training is a key component of strategy, not just for HR, but at the enter- 
prise level, where it can even serve as a competitive differentiator in a market. 


Does education have those impacts? We’d argue no, or at least not remotely to the same extent. We 
hire people based on their education and experience, whereas training fills in and insulates the gaps in 
knowledge and skills, and those gaps can and will differ across jobs and industries. Another way of think- 
ing about this is whether your graduate education in I-O prepared you with everything you needed to 
know for a job in academia or practice. Did you already have the people skills to behave like a profes- 
sional across all the situations you’d face? At hire, did you understand how to navigate the processes 
and systems you’d interact with in your job? Were you prepared to handle the shifting social dynamics 
and demands on your time? To varying degrees, some of us did, but it’s impossible that this is true of 
everyone in the l-O community. We had the education, but we needed training—we had to learn more, 
and what we needed to learn was patently job relevant in nature. This is precisely the challenge with 
which all L&D functions grapple. So, although training and education both involve learning, the stakes 
for training (and for bad training) have consequences at the employee and organizational levels in a way 


that education usually does not. Conflating the two things together may seem harmless (and might be in 
some situations), but if their repercussions were the same there wouldn’t be two distinct industries that 
focus on learning and both personal and professional development. 


Conclusion 
The point is that although L&D professionals can learn a lot from our field, we also have a lot to learn about 


learning. We've discussed the relatively limited focus on training in the I-O field in previous columns (e.g., 
Whelan & DuVernet, 2017). Where I-O practitioners are well-represented in selection, change management, 


and other areas, L&D is a critical part of organizational functioning that deserves more attention from the I-O 
community. You can hire the best people, based on complex statistical models and validated preemployment 
tests, but nobody comes into a job knowing everything they have to learn. 


We originally conceived of this column as a stimulus for “more thought and focus on the ways in which I- 
Os can contribute to the L&D space, with the ultimate goal of observing [an increased] number of I-O 
articles, research studies, and collaborative applied projects related to training” (DuVernet & Whelan 


2016). Unfortunately, if the session focuses of our annual conference are any indication, it appears 
there’s still much work to be done. With 11 sessions listing training as a primary focus area, this year’s 
conference features five fewer training related sessions than last year and continues a downward trend 
over time (see Figure 1 and SIOP Program Explorer 2008-2016 for more information). 
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Figure 1. SIOP Conference Session Listing Training as Primary Content Area Over Time 


Thus, as we close our tenure as columnists for TIP and reflect on the L&D journey we’ve traveled, we 
know that we’ve just scratched the surface of this need. We’ve provided information about L&D roles 
and the ways they are organized within companies; we’ve highlighted trends in training from both the I- 
O and L&D perspectives; we’ve hopefully busted a few myths without annoying too many people. Most 
importantly, we hope we’ve inspired you to learn more about learning (and development) to continue 


down the road toward more cohesiveness across our two complementary fields. Although we’re hang- 
ing up our hats, we want to stress how important it is for you to consider how you fit in this puzzle and 
how you might apply your skill set to contribute to L&D. As stated by Gary Latham (2009): 


It is we academics/scientists who provide the theory and empirical data that enables we practition- 
ers to differentiate ourselves in the marketplace from, and make ourselves invaluable to, decision 
makers in the public and private sectors. 
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In this issue, we continue on our exciting new development for the International Practice Forum! With 
Walter Reichman (OrgVitality) and a number of I-O psychology practitioners and academics, we explore 
the ways in which “Industrial-Organizational Psychology Helps Heal the World.” Through a series of arti- 
cles, we present real and actionable ways in which I-O academics and practitioners have an impact in 
innovative and creative ways and how they have been helping to heal the world! 


Using Industrial-Organizational Psychology to Facilitate Restorative Change 
With a State Prison Population 


Brian D. Cawley 
Calvin College 


Laura J. Shankster-Cawley 
Talent Asset Advisors 


Nicole Karl 
Calvin College Center for Social Research 


Statistics on crime and the prison system in our country are sobering. Headlines and news feeds are 
filled each day with dire stories of crimes and abuses that have a dramatically negative influence on 
our society. For example, here are just a few grim stats from a recent report produced by the United 
States Bureau of Justice?: 


e Approximately two-thirds (67.8%) of released prisoners were arrested for a new 
crime within 3 years of release, and three-quarters (76.6%) were arrested within 
5 years. 

e More than a third (36.8%) of all prisoners who were arrested within 5 years of re- 
lease were arrested within the first 6 months after release, with more than half 
(56.7%) arrested by the end of the first year. 

e A sixth (16.1%) of released prisoners were responsible for almost half (48.4%) of 
the nearly 1.2 million arrests that occurred in the 5-year follow-up period. 


As |-O psychologists, we invest our time and talents in careers studying the intersection of people and 
work. We are specialists trained to observe and diagnose behavior within organizational contexts to 
drive organizational performance. Certainly, a fair degree of our work involves those preparing to enter 
new jobs and even students early on in their vocational journeys. Our ability to measure individuals’ 
unique constellations of KSAOs undoubtedly helps people find work in which they can succeed and, in 
turn, helps contribute to the success of organizations at large. However, our target group is typically 


those already actively engaged in the job search process with relatively easy access to information about 
organizations, jobs, and themselves. 


Clearly, there are populations of potential employees that we are unable to reach through traditional 
avenues; one such population gaining increased attention is that of incarcerated individuals. As a soci- 
ety, we are slowly engaging in more constructive conversations about how we can collectively embrace 
the restorative power of work to not only ease societal reentry of former offenders but also to reduce 
recidivism rates. However, the process is complicated by a web of legal, cultural, and societal limita- 
tions—not the least of which is employers’ unwillingness to seemingly “take a risk” on someone with a 
felony conviction. 


As academic practitioners, we are continually challenged by our students to articulate not only what 
makes work scientifically “good,” but also to discuss the broader purpose of work in our lives. In our 
journey of understanding the inherent restorative power of work, we have been challenged by the 
thought that for some in our society, the concept of meaningful employment seems unattainable. This is 
due to the figurative and literal bars that have resulted from various life choices and circumstances, and 
create a real barrier for those individuals who are ready to take their place in the world. While we were 
asking ourselves questions surrounding the restorative power of work, we learned of some ground- 
breaking work being done by the professionals of the level Il, medium security Richard A. Handlon Cor- 
rectional Facility in lonia, Michigan, less than an hour away from our classroom in Grand Rapids. 


In their own words, their unique Vocational Village Program “is a first-of-its-kind skilled trades training 
program that aims to provide a positive learning community for prisoners who are serious about com- 
pleting career and technical educations.”? Prisoners are not only enrolled in full days of training and 
classroom instruction that mimic a real work environment, they also receive state and nationally recog- 
nized certifications in their trade upon successful completion of the certification process and graduation. 
Given the nationwide shortage of employees in the skilled trades, the prison has found employers 
throughout the state willing to recruit from the offender population and the placement rate of gradu- 
ates is currently over 70%. The offenders are filling a vital need for the employers, but the transition to 
gainful employment is not without pitfalls for many of the graduates. Although the employers have re- 
ported that the graduates’ technical skills are solid, many of the Vocational Village Program graduates 
lack, or are unaware of how to demonstrate, general employability competencies (e.g., dependability 
and reliability, adaptability and flexibility, initiative, professionalism, teamwork), which makes the on- 
boarding and probationary period challenging. 


We approached the prison leadership with a proposal to create and administer a Developmental Assess- 
ment Center for Vocational Village Program to students who are nearing their graduation and parole 
dates. The center provided an assessment of their general employability competencies and produced 
development reports for each offender to use as they prepared not only for employment but also for 
the start of a career in the skilled trades. For some of them, this was their first time achieving employ- 
ment in something other than entry-level, part-time jobs. The process was developed and administered 
by 30 upper-level undergraduate students enrolled in an industrial-organizational psychology course at 
Calvin College in Grand Rapids, Michigan, under the direct supervision and guidance of two professors. 
In addition, a team of experienced assessors from the business and professional community were re- 
cruited to provide the offenders with “real-world” feedback while also giving the employers insights into 


the program and the competency levels of this cohort of graduates. This way, they could further com- 
municate the benefit of the program to additional potential employers of the graduates. 


The professors and research assistant who served as the project leaders began the planning process 2 
months prior to engaging the college students in the work. The project included: 


1. Needs assessment with instructors, employers and a review of the literature to identify the assess- 
ment competencies; 

2. Development of an assessment protocol consisting of psychometric assessments and feedback via 
the jobZology® platform (values, interests, personality, and workplace preferences), a team project 
solving simulation, and a structured competency based interview; 

3. Development and delivery of participant preparation materials; 

4. Virtual training of 44 assessors; 

5. Administration of the center to 32 offender participants within a 3-hour assessment window physi- 
cally inside the prison; 

6. Writing of reports containing developmental feedback on each competency for the participants. 


The direct benefits to the offenders are quite typical of any assessment process but with a unique lens. 
For some, their familiarity with formal hiring assessments was minimal, but now they gained the oppor- 
tunity for recruitment by several international corporations in Michigan who employ state-of-the-art hir- 
ing systems. Thus, the men were introduced to new assessment tools, such as jobZology®, that they may 
potentially encounter in the regular recruiting process and were given the opportunity to “prepare/prac- 
tice” their performance before the event. The preparation process targeted their attention on compe- 
tencies (beyond technical skills) that are important to employers. Although the instructors and prison 
staff reinforce many of these competencies, it easy for the offenders to “brush them off” and not take 
the feedback seriously. Therefore, the entire conversation leading up to the actual assessment became 
an important source of development. The event itself generated a tangible energy among the offenders 
as they were able to engage with one another and provide support to one another during their orienta- 
tion time, team problem-solving exercise, and when escorted to their individual interview rooms. The 
assessment evening started with a sense of quiet anticipation and ended with engaging conversations, 
laughter, and, for many, an eagerness to receive and read their development report. 


As with any service learning initiative, our college students received many benefits from their participa- 
tion as well. First, they were able to engage in a multicultural experience involving interactions with indi- 
viduals from varying cultural perspectives within the unique institutional culture of a prison. Although 
our student body consists of roughly 30% international and/or students of color, the interactions at the 
prison gave them exposure to the unique culture that develops within correctional facilities. Throughout 
the project, students reported that their preconceptions of prison life and incarcerated individuals had 
been radically different from the reality they experienced. It was typical for students to be somewhat 
reserved upon entering the prison, but almost without exception, we witnessed radical transformations 
as they engaged in conversations about shared experiences such as the challenges of finishing class- 
work, preparing resumés, contemplating vocational choices, and the process of job hunting. Second, 
they not only learned the theories supporting assessment and development, and discovered the chal- 
lenges of translating such theories to the administrative realities facing organizations, but they were 
able to build relationships with people they were serving. Suddenly the long hours of creating behavior- 


ally anchored rating scales (BARS) and designing a group problem solving activity that could be con- 
ducted with “prison-approved supplies” became easier because they did not want to let the offenders 
down. Several students even decided to volunteer as tutors or interns at the prison because of this pro- 
ject. Finally, they were given a unique perspective on the restorative power of work as they saw first 
hand the way the men “lit up” when answering questions about their work, when they proudly dis- 
played the results of their training, and when the men talked with utter astonishment about the fact 
that for the first time in their lives they may actually obtain benefits like healthcare and even a 401k. 
The students were struck that these “standard benefits” that many college graduates take for granted 
represented real life-changing potential. 


Of course, the offenders and our students were not the only ones transformed. As project leaders and l- 
Os within both academia and applied settings, we have been further convicted that we have been so 
busy studying the intersection of people at work that we have looked past the restorative power of 
“work” itself. Engaging with individuals for whom the opportunity to work has been taken from them 
and is now once again within reach has allowed us to witness the immense power work has to restore 
purpose and dignity. While using our training to help restore the lives of our fellow citizens through de- 
veloping their skills for re-entry into the workforce, we have to take a step back and first remember and 
appreciate the restorative power of work for us all, and what that means for us all to flourish in our 
work environments. 


Do you know of someone who is using I-O psychology to heal the world? 


WE NEED YOU AND YOUR INPUT! We are calling upon you, the global l-O community, to reach out and 
submit your experiences for future columns. Give us your insights from lessons learned as you help heal 
the world. 


To provide any feedback or suggestions on the International Practice Forum, please send an email to the 


following address: lynda.zugec@theworkforceconsultants.com 


Notes 


1 Recidivism of Prisoners Released in 30 States in 2005: Patterns From 2005 to 2010-Update. 
https://www.bjs.gov/index.cfm?ty=pbdetail&iid=4986 
2 http://www. michigan.gov/corrections/0,4551,7-119-33218 75514-389694--,00.html 
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I’m pleased this issue to welcome a guest coauthor, one of the founding members of the Global 
Task Force for Humanitarian Work Psychology and the first Chair of the Global Organisation for 
Humanitarian Work Psychology (GOHWP), Dr. Ishbel McWha-Hermann. This is not Ishbel’s first 
time in TIP’s pages—far from it! Readers who have followed this column have seen her name in 
the byline in the past, and mentions of her important work both in the “Spotlight” column and 
in reports from SIOP’s UN team. She has lived and worked all over the world, and through her 
work has developed a perspective on issues related to work in the humanitarian sphere that has 
helped to significantly expand the borders of l-O psychology. 


One of her recent undertakings is Project FAIR, which she described last summer on the GOHWP 
website. Because this is such an important project, | asked her to write a piece on it for TIP, and 
she graciously accepted. She first provides some background on Project FAIR and its precursors, 
then answers several questions TIP readers might have about the project. With that, | give you 
Dr. Ishbel McWha-Hermann! 


Background 


When international nongovernment organizations (INGOs) send staff abroad they traditionally 
send them on an international reward package, which includes pay benchmarked to global pay 
data, and different benefits and allowances to those given to national staff. Historically, those sent 
on international packages were from the headquarter country, or from other similar (higher in- 
come) contexts, so the international package was used to attract skilled people into these roles, 
and to reflect the market from which they were being recruited. Times have changed now, 
though; for example, we see more regional-level and South-South recruitment, as well as increas- 
ing skills amongst national staff, bringing the relevance of these big packages into question. 


The focus of this column is on Project FAIR, which builds on previous research that examined the 
psychological impact of reward packages offered to national and international employees of 202 
organizations in six countries. In the original project (Project ADDUP) we found dual salaries (i.e., 
where national and international staff have different remuneration scales) had a negative im- 
pact on employees, but particularly on host country national employees in terms of motivation, 
satisfaction, and thoughts of leaving the organization. 


One of the key goals following Project ADDUP was to reach out to INGOs and share the results 
of the project, in the hope we might facilitate some change in how staff are rewarded. What we 
found was that human resource (HR) and reward managers didn’t need evidence of the issues 
with dual salary systems. Rather, they needed evidence-based alternatives to these systems. 
Our findings aligned with what managers were hearing from their staff feedback, and they felt 
frustrated about the situation, but powerless to change it without research into how it could or 
should be done. As we spoke to organizations about the issue, we also discovered that some 


had started trialling different reward options, but few were speaking to one another about what 
they were doing and how it was working. 


Project FAIR was born from the desire to gather these options together and begin to build an ev- 
idence base of different reward options that are being used by organizations. We wanted to 
gather stories from organizations about what works and what doesn’t work, as well as the chal- 
lenges faced in the process. HR and reward managers could then use this information as a basis 
for making decisions about how to set up rewards in their own organizations. We undertook 18 
qualitative interviews with HR and reward managers from 15 INGOs of varying size and scope, 
and developed some insights into the different approaches to reward that are being under- 
taken, as organizations try to find ways to make their reward systems fairer. A full report on the 
project findings is available on the website www.project-fair.org. 


How did you get involved in Project FAIR? 


Having worked in the NGO sector in both India and Cambodia prior to doing my PhD, | was 
drawn to issues of intergroup dynamics within development work, and the complicated role of 
social identity and social dominance on the long term prosocial goals of international aid and de- 
velopment work. Disparate salaries between national and international employees are one clear 
source of dominance and potential conflict for employees. | began looking at the psychological 
impact of dual salaries during my PhD with Stu Carr at Massey University, and continue to focus 
on this in my own research at the University of Edinburgh, with a particular focus on organiza- 
tional justice. 


What have been some of the key outcomes of the project? 


Of the 15 organizations involved in the project, three have already eradicated the dual salary 
system, and a further five have developed what we have called a ‘hybrid’ system, where they 
have taken steps toward reducing the gap between national and international reward, but still 
retain a dual system of some kind. Examples of this might be putting international employees on 
headquarter packages, rather than international packages, or matching executive salaries for all 
employees regardless of nationality. We were surprised to find such a lot of variability between 
the organizational approaches. However, while we found that organizations are taking different 
approaches to developing fairer systems there are some strategic issues that are commonly ad- 
dressed before making changes, for example, deciding on their mobility strategy, commitment 
to localization, and examining their total reward package. 


A published report of the full findings is available on the project website, along with six organiza- 
tional case studies. We are really excited about being able to produce these case studies as open 
access resources. We’ve already had a lot of great feedback from INGOs that they are really 
helpful for enabling them to think about how they might address the topic of fair reward, and to 
use as tools to advocate for change within their organizations. Side note for academic readers - 
they are also a great resource for teaching! 


In addition to the project outputs, we ran a full-day interactive workshop with HR and reward 
managers in London, as well as three webinars for audiences from different parts of the globe. 
There has been a huge amount of interest in the issue, perhaps (at least in part) because of the 


pressure on the INGO sector from international agreements like the Charter4Change (which fo- 
cuses on the importance of localizing roles in INGOs), and the UN Sustainable Development 
Goals (SDGs) (Goal 8 specifically aims to have equal pay for work of equal value by 2030.) 


Throughout all the interactions with practitioners the important role of academic researchers in 
facilitating knowledge sharing on a sensitive topic has been emphasised. As external and objec- 
tive individuals we enable organizations to talk about reward-related issues and challenges they 
face and brainstorm ideas, as well as share lessons learnt, but to do so in an anonymous way. 


What have reactions been to your work, from the various constituencies involved in the inter- 
national aid sector? 


The project has been welcomed by most major INGOs, some of whom we have now developed 
longer term relationships with and plan to engage in ongoing research on the topic. There is a 
general feeling within the INGO sector that for systemic change like this to occur organizations 
need to move together, not in a pay-fixing, prescriptive way, but in a collaborative way that 
helps to move the sector toward paying in a way that reflects shared goals. This might be, for 
example, an agreement to build principles of fairness into their reward systems (whether this be 
based on equity, equality, or need). Academic researchers and projects like this are key to sup- 
porting that. 


What kinds of challenges has the project team encountered in doing its work? 


The biggest challenge was related to the emotive and sensitive nature of the topic. We uncov- 
ered some excellent examples and cases where organizations were doing innovative and crea- 
tive things to underpin their reward policies with fairness, but many of them we are not able to 
discuss in detail for reasons of confidentiality. Organizations were often happy to share with us 
their strategies, but only under condition of anonymity, even when their strategies were among 
the most progressive in our study. For example, of the 15 organizations we interviewed, three 
have revised their reward packages to eradicate the dual salary system, and have now put all 
employees on a single scale (which is often based on a blend of national and international salary 
benchmarking data). Of those three organizations, only one was willing to be profiled as a case 
study, or even to be named. This hesitancy to discuss how people are being rewarded contrib- 
utes to a general feeling within the sector that there is much less movement toward addressing 
reward disparities than is actually the case. 


What do you see as the next steps, either for Project FAIR or beyond the current project? 


There is so much we as researchers can do to contribute to positive change in the context of 
managing human resources in INGOs, in particular there is a relative lack of research examining 
the experiences of host country national employees. Given the importance the international aid 
and development sector places on building national capacity, this might be even more im- 
portant in this sector than in others. In terms of reward fairness, we are currently in the process 
of developing a toolkit for organizations to use as they work through the process of considering 
change in their reward systems, and we are engaging in longitudinal evaluation of changes in 
some organizations. From a research perspective we are exploring what fairness actually means 
for reward in INGOs (this might be a context where need is more important than equality or eq- 


uity, for example). I’m also planning a project that explores employee identity, reward expecta- 
tions, and motivation for working with INGOs, to help to provide a clearer picture of the people 
working in the sector. Finally, I’m working to develop company sponsored dissertations for MSc 
students at my university to work on projects defined by INGOs. This is a win-win for students 
and INGOs, and is a lot of demand from students for prosocially-oriented work. 


Beyond the project itself, we need to find ways to actively contribute to the UN SDGs -there is a 
wealth of opportunity for l-Os to get involved with this, and an interest from INGOs and multilat- 
eral organizations to use our skills to help them. The SDGs are a further justification for why they 
should do so. SIOP is already making great inroads into this through the UN team, but there are 
other options in our own research agendas to align with the SDGs and therefore to make contri- 
butions individually too. 


Do you have any advice for TIP readers who might want to get involved in working with NGOs? 


My number one piece of advice is to prioritize relationships. People move around a lot so it can 
be a challenge to retain collaborations with organizations that are long term, but building rela- 
tionships and becoming part of the network enables a broader connection with the sector. The 
ability to be mobile is also useful, and in particular taking the opportunities to meet people in 
person is really valuable. 


On the topic of relationships, if you are interested in getting involved in some of the work of the 
Project FAIR team, please don’t hesitate to get in touch (ishbel. mcwha@ed.ac.uk)! 


Is there anyone you particularly want to recognize for their work on Project FAIR? 


This project has been a true collaborative effort, and is underpinned by a fantastic team spread 
around the globe — though our geographic dispersion means scheduling meetings is often a chal- 
lenge! Working alongside me at the University of Edinburgh are Dr Jakov Jandric and Emily 
Cook-Lundgren, and we are very lucky to receive ongoing expert advice from Prof Stu Carr at 
Massey University in New Zealand. We are also joined by colleagues from two practitioner or- 
ganizations — firstly, Sam Wakefield from CHS Alliance, a network organization focused on the 
humanitarian and development sector, and secondly, Curtis Grund from Birches Group, who of- 
fer expert compensation benchmarking and advice to the NGO sector. 


Thanks so much, Ishbel, for taking the time to share your thoughts and experiences on Project 
FAIR with TIP’s readers! 


And speaking of the readers, if there are topics within the humanitarian sphere that you want to 
know more about, send me an email, either direct (mullins@xavier.edu) or by reaching out to 
the GOHWP executive board through our website (http://www.gohwp.org). We know that 
many of you are engaged in interesting and important work in the humanitarian sector, and 
would love to help you get the word out! 


The Results Are In! Updated Alternative I-O Graduate Program Rankings 
Nicholas P. Salter, Joseph A. Allen, Allison S. Gabriel, Loren Naidoo, and David Sowinski 


In the summer of 2016, we issued a Call for Proposals to submit unique and innovative methodologies to 
rank I-O graduate programs. In response to this, many projects were proposed to us. After much hard 
work (and the broader SIOP community’s help), the five selected projects have been completed. Each of 
these five papers are included in the current issue of T/P. We believe each of these papers will be an im- 
portant contribution to our field and will guide individuals in the future — as well as generate much 
thought and discussion about the state of our field and the programs that educate the future of IO Psy- 
chology. In particular, we hope that these rankings will result in graduate programs examining them- 
selves and thinking about ways they excel, as well as areas they could improve. Additionally, we hope 
that future undergraduate students applying to I-O programs will use these rankings, not to determine 
what the “best” programs are, but which programs are the best fit for them. 


Project Description and Findings 


The original goal of this project was to highlight alternative ways of ranking (and more generally: exam- 
ining) |-O graduate programs. Previous rankings primarily focused (though not exclusively) on research 
productivity. Although this is an important index of graduate program success, it is not the only marker. 
Other ways of measuring the strengths of graduate programs are difficult to operationalize as well as 
execute. To this end, the current five projects answered this call to determine nontraditional ways of 
determining the strengths of graduate programs, as well as to compare schools against each other. We 
saw this project as an opportunity to widen the scope of how we as a field determine “success” in grad- 
uate programs, and to generally celebrate the various strengths different graduate programs offer. 


The process through which these projects were completed sought to strengthen each project as much 
as possible, as well as ensure that the rankings reflected the included graduate programs as much as 
possible. First, authors submitted their proposals to the reviewer committee (the authors of this paper). 
The reviewer committee was carefully composed to ensure diversity of program types, practitioners and 
academics, and those with graduate program education experience were included. The reviewer feed- 
back was incorporated into the projects, and then the authors were given approximately one year to 
collect their data and write their manuscripts. Once again, their manuscripts were sent to the reviewers 
for feedback, and revisions were made. Through this process, multiple people were involved to ensure 
the quality of the projects were as high as possible. 


The first two papers examine master’s programs, an area understudied by previous graduate rankings. 
Vodanovich et al. (2018) examined objective, quantifiable indicators of master’s program success, such 
as how much applied experiences students in the program receive, how much faculty are involved with 
applied work, job and internship placement rate, and number of courses students take in various topics. 
To gain this information, the authors of this project surveyed master’s program directors. The second 
project, conducted by Acikgoz et al. (2018), also examined master’s programs but instead focused on 
student and alumni perceptions of programs. The inclusion of the alumni perspective made this project 
especially unique; master’s program directors were asked to forward the survey to recent alumni who 
had graduate within the past 5 years. These respondents helped give a different view on the programs 
than other respondents, thus adding a more nuanced understanding of program quality. 


The third paper, authored by Landers et al. (2018), examined research productivity through a different lens than 
has been used previously. Instead of solely looking at the quantity of publications and/or conference presenta- 
tion, this paper examines the interdisciplinarity of research output coming from graduate programs. This paper 
looks at how often faculty from gradute programs are publishing papers outside of the traditional core l-O jour- 
nals, which provides an interesting way of thinking about scholarly output especially in addition to the previ- 
ously published graduate rankings that focus on the quantity of publications and presentations. 


The next paper, Howald et al. (2018) compared student perceptions of particular aspects of their programs with 
how subject matter experts in the field rated these aspects. The aspects they examined related to applied, 
teaching, and research developmental opportunities. The subject matter experts they surveyed in their study 
was the broad SIOP community; SIOP members rated the importance of various developmental opportunities 
graduate students may receive, and this was compared to whether or not current graduate students perceived 
these developmental opportunities to be available for them. Similarly, the final paper by Roman et al. (2018) 
also examined student perceptions of graduate programs. This paper looks broadly at multiple aspects of per- 
ceived program quality, including funding opportunities, class offerings, and general program culture. 


In sum, the set of papers included in this volume cover a wide range of rankings. These include rankings 
of masters and doctoral programs from a variety of angles, such as students’ perceptions as well as 
more objective indicators (e.g. interdisciplinary research productivity). The hope is that, in addition to 
traditional rankings, these additional perspectives provide more insight into the strengths and growth 
areas of the educational environment of IO Psychology across the field. 


Caveats and Conclusions 


Overall, we are pleased with the results of these projects and are excited to share the results with the SIOP 
community. A few caveats and considerations should be mentioned, though. First, response rate limited many 
of these projects; programs could not be included if there was not enough data on it. For some of the projects 
(i.e., those that averaged ratings of multiple people representing the program), a small response rate meant 
that the program was omitted from the ranking (in an effort to protect the confidentiality of those who did re- 
spond). A casual reader might see that a school was not included and implied that the school was poorly 
ranked, but this is not the case. Please keep this in mind as you are reading and interpreting the results. 


Also, it is important to remind the reader that although we believe these rankings provide unique and inter- 
esting ways of operationalizing and examining various program strengths, these are not the only methods of 
doing so. The premise of this endeavor was that there are multiple ways of ranking graduate programs not 
just research productivity. Therefore, it would be disingenuous for us to say that we have created a definitive 
list of all the ways graduate programs can excel. We acknowledge that there are many more ways graduate 
programs could have been analyzed and ranked, and we hope that these five papers will spark discussions 
among I-O psychologists about other program aspects that should be emphasized. 


In fact, we hope that these papers spark multiple discussions among individuals in our field, not just 
about what else could have been examined, but just generally what people think about these rankings. 
What do you agree with? What do you disagree with? If you would like to join in on the discussion, we 
encourage you to attend our session at the 2018 SIOP conference entitled “Where Do We Stand? Alter- 
native Methods of Ranking I-O Graduate Programs.” This session will be on Friday, April 20 from 11:30 — 
12:50 in the Gold Coast room. The format of the session will be that the project authors will give a brief 
overview of their findings, the reviewer committee will discuss the findings, and then the audience will 
be open to ask questions and offer their thoughts. We hope to see you there! 


Ranking I-O Master's Programs Using Objective Data From I-O Coordinators? 


Stephen J. Vodanovich, Valerie J. Morganson, & Steven J. Kass 
University of West Florida 


Past I-O psychology program ranking surveys have differed in a number of respects such as sample type 
and size (e.g., faculty, students), type of data collected (e.g., objective, subjective), and number of pro- 
grams ranked, the majority focused on ranking doctoral programs based on research productivity 
(Beiler, Zimmerman, Doerr, & Clark, 2014; Gibby, Reeve, Grauer, Mohr, & Zickar, 2002; Oliver, Blair, 
Gorman, & Woehr, 2005; Payne, Succa, Maxey, & Bolton, 2001; Winter, Healy, & Svyantek, 1995). An 
exception is the research of Kraiger and Abalos (2004) who surveyed masters and doctoral students with 
a focus on nonresearch factors (e.g., instruction quality, faculty support, funding, class size). 


Given that previous survey research focused on doctoral programs and research output, our proposal ex- 
tended current rankings by concentrating on master’s level programs and on applied experiences. Addi- 
tionally, we relied upon objective data. The highlighting of applied experience seemed warranted given 
that applied skills are arguably more pertinent to ranking MA/MS programs as compared to assessing doc- 
toral training. Indeed, past research on master's I-O programs have discussed the vital role of application 
of classroom material through such mechanisms as consulting projects and internships (e.g., Hays-Thomas 
& Kass, 2003; Lowe, 1993; Kottke, Shoenfelt, & Stone, 2014; Schneider, Piotrowski, & Kass, 2007; Shoen- 
felt, Kottke, & Stone, 2012; Shoenfelt, Stone, & Kottke, 2013). Also, the focus on application is consistent 
with SIOP's Guidelines for Education and Training, which recognizes that graduates of master’s programs 
are more likely to be consumers of I-O research (e.g., using the literature to solve work-related problems) 
than they are to be researchers themselves. We wrote items, a priori, to assess the following dimensions: 
(a) applied experience, (b) curriculum, (c) faculty experience/information, and (d) student accomplish- 
ments/information. We discuss each of these dimensions and their rationale below. 


Applied Experience 


The first, quintessential dimension—applied experience—reflects the degree to which students are en- 

gaged in various real-world opportunities to use their I-O skills. These include the availability of applied 

projects, internships, and the presence of a consulting unit. Such activities provide enriching applied ex- 
periences for graduate students (e.g., Byrne et al., 2014; Dickson & Mullins, 2016; Kottke et al., 2014). 


Curriculum 


Curriculum is inherently critical to ranking educational programs. The curriculum dimension included 
number of credit hours, I-O-related credit hours, time to graduation, proportion of students who gradu- 
ate on time, and course offering. Our assessment of I-O masters programs’ course offerings was based 
on past work that identified the importance of specific courses and competencies for graduate study. 
This work included the SIOP Guidelines for Education and Training in Industrial Organizational Psycholo- 
gy (2016) as well as other relevant research (e.g., Erffmeyer & Mendel, 1990; Tett, Walser, Brown, 
Simonet, & Tonidandel, 2013; Trahan & McAllister, 2002). 


Faculty Experience/Information 


In parallel to historical rankings that have assessed programs based on faculty research productivity (see 
citations above), faculty applied experiences and qualifications are a component of our ranking proce- 
dure. Faculty applied experience was included to help applicants evaluate the type of mentoring they 
can expect in a program. One might expect that faculty who themselves have engaged in applied expe- 
riences (e.g., consulting clients) would be better able to supervise students who are engaged in hands- 
on experiences (e.g., fieldwork, internships, projects) by modelling the expected behavior. This is the 
educational approach suggested in the SIOP Guidelines (2016) for building professional competence. 
Faculty experiences/information included faculty-to-student ratio, proportion of faculty who engage in 
consulting, and proportion of faculty who supervise consulting projects. Because scholarship is essential 
to quality applied intervention, we also included an item to reflect faculty research productivity. 


Student Accomplishments and Information 


The assessment of student accomplishments evaluates programs in terms of what applicants could ex- 
pect while matriculating (e.g., assistantships) as well as upon graduation (e.g., obtaining jobs). To the 
extent that strong students help attract and retain other strong students and build the reputation and 
human capital of a program, student accomplishments can be viewed as an indicator of program quality 
(cf. Schneider, 1987). Perhaps more intuitively, student accomplishments can be viewed as an outcome 
of program quality. Therefore, we decided to present program rankings on the student accomplish- 
ments/information dimension as well as an outcome of the other three factors that we measured. Stu- 
dent accomplishments/information included items concerning percentage of non-doctoral-bound stu- 
dents who obtain an I-O job within a year, active participation in |-O-related student chapters, assis- 
tantships, graduation rates, and presentations at conferences. 


Method 
Participants, Procedure, and Instrumentation 


Prior to administering the survey, the project was approved by the university’s institutional review 
board. After receiving the contact information for each of the 127 coordinators of |-O MA/MS programs 
from SIOP, we sent emails to each coordinator with a brief description of our research, informed con- 
sent, and a link to our survey in Qualtrics. We surveyed coordinators of terminal master’s programs as 
well as MA/MS programs contained within doctoral programs. Weekly reminders were sent to coordi- 
nators to increase the response rate. These updates targeted coordinators who had not yet begun the 
survey or whose survey responses were incomplete. Overall, coordinators were allowed approximately 
6 weeks to complete the survey. This extended time was provided because several items required coor- 
dinators to assess archival data (e.g., graduation rates) in order to provide accurate responses. 


Our final sample consisted of 69 completed surveys for a response rate of 54%.” Degree types included 
MS (47.8%), MA (44.9%), or other degree (7.2%; e.g., both MA and MS, MPS). The vast majority (92.8%) 
of respondents indicated that their program was located within psychology. Programs embedded within 
public institutions (68.1%) were more prevalent than programs within private institutions (31.9%). 
Whereas 69.6% of respondents indicated that they did not have a doctoral program, 30.4% did. The ma- 


jority of respondents reported having a face-to-face program (81.2%), but our sample also included hy- 
brid programs (10.1%) and exclusively virtual programs (8.7%). With the exception of two programs, re- 
spondents were located at universities within the United States.? 


The final survey consisted of 53 questions. The items were written and placed into the four dimensions 
noted earlier for ranking purposes: applied experience, curriculum, faculty expertise/information, and 
student accomplishments/information. Additional items assessed "demographic" program data that 
were not used for ranking purposes (e.g., faculty diversity, job tenure, private or public university). At 
the end of our survey, we provided a link to a page provided by a team of researchers at Appalachian 
State University who were conducting a different ranking survey for SIOP. 


The items on the survey were developed using several factors. One was the inclusion of items to collect 
program data that are regularly reported by SIOP via input from program coordinators, (e.g., number of 
l-O faculty, graduate employment, average program completion time). Other items were written to be 
consistent with previous surveys (e.g., Tett et al., 2013) and SIOP's 2016 Guidelines for Education and 
Training in Industrial-Organizational Psychology (e.g., curriculum coverage, applied focus for MA/MS 
programs). Finally, the items were reflective of those contained in previous program ranking surveys, 
such as applied experience, coursework, financial aid, and student research opportunities (Bulger, 
Horvath, & Zickar; Kraiger & Abalos, 2004). We originally planned to conduct a pilot study from a ran- 
dom sample of I-O coordinators to assist in item construction, but the tight timeframe for data collec- 
tion made this approach untenable. 


Because items varied in terms of response format (e.g., ordinal categories, continuous data), all re- 
sponses were scaled from 0 to 1 in order to sum items into category scores. For example, responses 
were assigned point values (e.g., 0 to 5) based on higher scores being indicative of greater quality (e.g., 
number of hours needed for internships, total I-O related hours required, total program hours, number 
of l-O faculty). Points were then divided by the number of options that were available on each item. 
Further, items that asked about proportions were divided by 100 (e.g., percent of I-O students who 
graduate, percent of students who do an internship). Items were averaged together to create category 
scores on which programs were ranked. Finally, category scores were combined to provide an overall 
ranking. Data cleaning was informed by content coding of qualitative write-in responses, referencing 
the SIOP program database, and in some cases, replying to emails from respondents. For instance, in 
order to enhance the accuracy and consistency of survey responses, we responded to several emails to 
clarify the meaning of various items. 


Results 


Means, standard deviations, and intercorrelations among dimensions are presented in Table 1. Pro- 
grams were ranked on each of the scale facets as well as overall. Top five rankings for each dimension 
are presented in Tables 2 to 5. As indicated earlier, we also regressed Student Accomplishments/ Infor- 
mation onto each of the other facets: Applied Experience (B = .29, p = .02, R? = .08), Curriculum (B = .29, 
p = .02, R? = .09), and Faculty Experience/Information (B = .35, p = .02, R? = .12). Each significantly pre- 
dicted Student Accomplishments/Information. The top ten rankings overall—an average across the four 
dimensions—is presented in Table 6. 


Table 1 
Means, Standards Deviations, and Intercorrelations Among Dimensions 


M | SD 1 2 3 4 

Applied experience 59 .20 

Curriculum .71 | .15 | .31** 

Faculty experience/information 52  .17 .28* 18 

Student accomplishments/information 66 16 .29* | .29* .35** 

Overall .62 | .11 .73** .63** | .67** .69** 

Note. *p < .01 **p <.001 

Table 2 

Rankings of Programs Top 20 Institutions/Programs for the Applied Experience Dimension 

Institution/program Ranking score (0 to 1) Z-score 

Minnesota State University, Mankato .94 1.83 

University of West Florida .93 1.77 

University of Detroit Mercy .90 1.61 

Middle Tennessee State University .90 1.60 

St Mary's University .90 1.58 

Southern Illinois University at Edwardsville .89 1.54 

San Francisco State University .88 1.51 

Saint Cloud State University .88 1.51 

Valdosta State University .87 1.43 
, Florida Institute of Technology .83 1.26 
, George Mason .83 1.23 
, William James College .82 1.20 
, Angelo State University .81 1.15 
, University of Texas at Arlington .78 97 
, Roosevelt University .76 89 
, University of Tennessee at Chattanooga 75 83 
, California State University, Long Beach 75 .83 
, Western Kentucky University .72 .67 
, Emporia State University 72 .66 
, Springfield College 71 .62 

Table 3 

Rankings of Programs Top 20 Institutions/Programs for the Curriculum Dimension 

Institution/program Ranking score (0 to 1) Z-score 

University of Tennessee at Chattanooga .94 1.50 

Middle Tennessee State University .90 1.25 

University of New Haven .88 1.13 

Appalachian State University .88 1.12 

California State University, San Bernardino .88 1.10 


Hofstra University .87 1.02 


Carlos Albizu University-Online 85 91 


Minnesota State University, Mankato .84 88 
Carlos Albizu University-Miami .84 85 
, South Dakota State University .84 84 
, lona College .83 77 
, San Francisco State University .82 10 
, Salem State University .82 .68 
, Xavier University .81 .67 
, St Mary's University .81 .66 
, University of Georgia .81 .64 
, Florida Institute of Technology .80 .61 
, University of Detroit Mercy .80 .60 
, Valdosta State University .80 .60 
, Lamar University .80 59 
Table 4 
Rankings of Programs Top 20 Institutions/Programs for the Faculty Experience/Information Dimension 
Institution/program Ranking score (0 to 1) Z-score 
Illinois State University .90 2.22 
Florida Institute of Technology .76 1.39 
University of Tulsa 75 1.34 
Appalachian State University 74 1.28 
University at Albany, SUNY 73 1.23 
Saint Mary's University 71 1.13 
Minnesota State University, Mankato 70 1.04 
George Mason 70 1.04 
Southern Illinois University at Edwardsville .70 1.04 
, Elmhurst College .70 1.02 
, Middle Tennessee State University .70 1.02 
, Eastern Kentucky University .69 97 
, Central Michigan University .69 .96 
, Colorado State University .68 .90 
, Radford State University .67 85 
, University of Oklahoma .67 85 
, Wayne State University .66 .80 
, Western Kentucky University .66 79 
, Xavier University .65 75 


, San Diego State University .64 .71 


Table 5 
Rankings of Programs Top 20 Institutions/Programs for the Student Accomplishments and Information 
Dimension 


Institution/program Ranking score (0 to 1) Z-score 
South Dakota State University 1.00 2.15 
Middle Tennessee State University .97 1.96 
Southern Illinois University at Edwardsville .94 1.76 
University of Minnesota Duluth .92 1.65 
Central Michigan University .87 1.34 
San Diego State University .87 1.31 
Saint Mary's University .86 1.23 
Lamar University 85 1.19 
Appalachian State University .84 1.14 
, Illinois State University .84 1.13 
, Minnesota State University, Mankato .84 1.13 
, Western Kentucky University .83 1.05 
, University of Central Florida .81 92 
, University of Nebraska at Omaha .80 89 
, Florida Institute of Technology .79 82 
, University of Oklahoma 79 81 
, Eastern Kentucky University .77 .68 
, Radford State University 77 .66 
, Indiana University Purdue University Indian- .76 59 
apolis 
, University of Tulsa 75 57 
Table 6 
Overall Rankings of Top 20 Institutions/Programs 
Institution/program Ranking score (0 to 1) Z-score 
Middle Tennessee State University .87 2.14 
Minnesota State University, Mankato .83 1.84 
Florida Institute of Technology .80 1.54 
Southern Illinois University at Edwardsville 77 1.33 
University of West Florida .76 1.23 
University of Tennessee at Chattanooga .76 1.19 
Saint Cloud State University .76 1.18 
Appalachian State University 75 1.15 
George Mason 75 1.10 
, Western Kentucky University 75 1.09 
, St Mary's University 74 1.06 
, San Francisco State University 74 1.05 
, University of Detroit Mercy 73 93 


, University of Central Florida 72 .90 


, California State University, San Bernardino 71 .80 

, Central Michigan University 71 76 

, San Diego State University 70 713 

, Eastern Kentucky University 70 12 

, Illinois State University .69 .64 

, Angelo State University .69 .63 
Discussion 


This project presents a contribution to SIOP and potential graduate students. Specifically, past program 
rankings have primarily assessed academic factors that are more relevant to doctoral-level education. 
However, a strength of the current study is that it focused on applied factors, which are more applicable 
to the mission of I-O master’s programs and more pertinent to graduates who are likely to become prac- 
titioners. Also, besides providing an overall program ranking, our results offer rankings on four specific, 
relevant program dimensions. This approach was a major focus of our survey, which was to provide 
prospective students with a useful way to view the quality of master’s programs on multiple dimensions. 
As such, this article provides students with a resource to help them choose the right program for them 
based on the dimensions they consider to be most important. That is, they can decide whether to focus 
on overall program rankings or emphasize specific program components (e.g., applied experience, cur- 
riculum). Our findings can also be used as a benchmark for I-O faculty who want to build or enhance 
their programs while also allowing them to emphasize key, positive programmatic features to appropri- 
ate audiences. 


Strengths and Limitations 


Our project has limitations as well as strengths. Ideally, formative measures should be identified and 
validated with criterion-related validity evidence using structural equation modeling (e.g., Diamantopou- 
los, Reifler, & Roth, 2008). However, our sample size and project scope did not permit such a validation 
effort. Regression results and facet intercorrelations, however, provide some evidence of criterion- 
related validity. The content of some items could be considered as being related to more than one cate- 
gory. This was especially true regarding the “applied experience” and “curriculum” dimensions. For in- 
stance, questions about practicums, internships, and the number of courses requiring formal presenta- 
tions were included under the “applied experience” category, although they can also be considered as 
part of a program’s curriculum. Also, as noted before, our student accomplishments/information di- 
mension contained program factors (e.g., availability of graduate assistantships) as well as program 
“outcomes” such as gaining employment after graduation and graduation rates. Our regression analysis 
partly addressed this issue by using student accomplishments/information as an outcome variable. Re- 
gardless of how this dimension is conceptualized, its contents are likely to be considered by students in 
evaluating the quality of I-O master’s programs. 


Our survey required answers to objective items. However, it is possible that some I-O coordinators may 
have interpreted the meaning of various questions differently (e.g., what qualifies as an internship or an 
applied project, the extent to which various topics are covered in the curriculum). Also, certain data 
may not have been readily available to coordinators (e.g., what percentage of graduates receive an l-O — 


related job within a year, what percent of I-O students presented papers at conferences). If so, the ac- 
curacy of the responses could be an issue. Regarding the latter point, |-O programs would benefit if 
their coordinators established mechanisms to track and assess program quality and indicate areas for 
improvement. Finally, some programs were too new to adequately answer many of the survey questions 
(e.g., graduation rate). Therefore, they were not included in our rankings. 


A strength of our project was its focus on objective, quantifiable factors rather than potentially biased 
opinions of satisfied (or dissatisfied) respondents. We tried to include data that most program coordina- 
tors would likely be collecting as they assess and track their own programs. Finally, at 54%, our response 
rate was quite respectable, thus adding to the representativeness of our findings. With more and more 
programs competing for qualified students, we expect that future updates to the rankings will include data 
from all available programs. The survey results could help guide decision makers (e.g., curriculum, applied 
experiences) in taking steps to improve their individual programs. It is our hope that the discrete, objec- 
tive information contained in our results will be combined with additional subjective data that will allow 
prospective students to identify programs that best fit their interests and career goals. 


Notes 

1 The authors wish to thank Skye Evans and Michael DeNoia for their assistance on this project. 

2 Eleven surveys were partially completed and were not included in the analyses. 

3 In one case, an international university’s response had to be omitted for the curriculum dimension because they 
used a points system rather than a credit hour system; no conversion was available. 

* Data were extensively cleaned prior to analysis. Additional detail on specific data cleaning decisions is available 
upon request from the authors. 
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Appendix A 
Items Used to in Rankings by Facet 


Applied Dimension 
1. Responses were combined on the following two questions: 
a. Does your I-O program contain a formalized, applied internship within your curriculum? 
b. How many hours are required to successfully complete the internship? If variable, provide an 
average. Type in number of hours. 

2. Typically, what percent of students perform an internship? 

Does your program allow for students to enroll in a practicum? 

4. Does your program have a designated unit (e.g., consulting clinic, center) to acquire consulting con- 
tracts and/or grants? 

5. How many courses in your program, including an internship, if applicable, involve students conduct- 
ing applied projects (e.g., job analysis, training programs, organizational development) outside of 
the classroom? 

6. How many courses in your program require formal presentations (group or individual) designed for 
applied audiences? 


2 


10 


Curriculum Dimension 


1. 


Responses were combined on the following two questions: 

a. How many total credit hours are required for your |-O MA/MS degree? Please indicate in semes- 
ter hours (1.5 quarter hours = 1 semester hours). 

b. Given the number of hours in your program, what percent of MA/MS students graduate in the 
expected timeframe (e.g., “on time”)? 

How many I-O-related hours (including research methods and statistics) are required for your I-O 

MA/MS degree? Please indicate in semester hours (1.5 quarter hours = 1 semester hours). 

To what extent are the following topics covered in your program? Use the guidelines listed below in 

providing your answers: (0 = never: not covered at all, .5 = somewhat, 1 = extensively). [job analysis, 

personnel recruitment/selection, training and development, performance appraisal, job evalua- 

tion/compensation, employment law, work motivation, work attitudes, work groups/teams, leader- 

ship/management, judgment/decision making, organizational development, organizational theory, 

work/family, work stress, human factors, consulting/business skills, workforce diversity, workforce 

aging, individual differences in the workplace] 


Faculty Information/Experience 


1. 


Faculty to student ratio 

a. How many l-O faculty teach in your l-O program? (three-quarter appointments count as .75; 
half-time appointments count as .5; one-third appointments count as .3; Do not count adjunct 
instructors) 

b. This was divided by responses to the following question: What number of students typically en- 
ter your l-O program each year? 

What number of I-O faculty in your program have worked on a consulting project in the past 5 

years? (This was divided by the number of I-O faculty who teach) 

How many I-O faculty have supervised l-O students on external consulting projects? 

(This was divided by the number of students enrolled annually) 

How many total articles have been published by your I-O faculty in refereed journals from 2012 to 

2016, including “in press” articles? 


Student Accomplishments/Information 


1. 


> 


Of the graduate students who do not pursue doctoral degrees, what percentage obtain work in an 
l-O-related job within a year after graduation? 

What percent of current I-O graduate students are active participants in l-O related student chapters 
(e.g., SHRM, ATD, IOPSA)? 

What percent of current I-O graduate students received assistantships? 

Of those who enroll in your program, what percent of your I-O students graduate? 

What percentage of your MA/MS students have presented a paper or poster at a regional, national, 
or international conference in the past 5 years? 
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There is little dispute that reputation is a factor in a prospective student’s decision to apply to a specific 
graduate program in industrial-organizational (I-O) psychology. Consequently, there have been many 
attempts at ranking graduate programs. However, the vast majority of these have examined doctoral- 
level programs and have used research productivity as the main criteria (e.g., Beiler, Zimmerman, Doerr, 
& Clark, 2014; Gibby, Reeve, Grauer, Mohr, & Zickar, 2002; Oliver, Blair, Gorman, & Woehr, 2005; Payne, 
Succa, Maxey, & Bolton, 2001; Winter, Healy, & Svyantek, 1995). Especially for master’s level graduate 
programs, subjective program experience may be an important consideration for prospective students 
when evaluating the desirability of a graduate program. An example of this kind of program evaluation is 
Kraiger and Abalos (2004), but it has been more than a decade since the publication of their results, and 
an update is long overdue. This project attempts to remedy past neglect and provides rankings of mas- 
ter’s level I-O graduate programs using subjective ratings provided by students and alumni (a companion 
report in this issue, Vodanovich, Morganson, & Kass, provides a ranking of master’s degree programs 
based on objective criteria). 


Our project was prompted by a call for proposal by SIOP (Salter, Allen, Gabriel, Sowinski, & Naidoo, 
2016). A major goal of this effort was to identify more comprehensive ways of ranking programs using 
the criteria that would be important to prospective students. We took this as a call to further develop 
subjective criteria to rank master’s level programs and decided on three main indicators of program 
quality that would be evaluated by each program’s current students. First, there has been a wide range 
of research supporting the importance of culture in educational success (Pritchard, Morrow, & Marshall, 
2005; Tichnor-Wagner, Harrison, & Cohen-Vogel, 2016) as well as business success (Sirota, Mischkind, & 
Meltzer, 2008; Xenikou & Simosi, 2006). Similarly, we believe having a supportive and warm culture in 
graduate school is important in creating a superior professional development experience. Accordingly, 
the first subjective indicator is program culture. Second, we measured program resources via students’ 
perceptions of resources such as program faculty, breadth of program curriculum, funding opportuni- 
ties, and availability of academic and professional mentoring. This offers a more in-depth examination of 
resources beyond typical research and faculty availability. Finally, the third indicator of program quality 
examined in this study is student satisfaction. We believe that a good program results in a high level of 
overall satisfaction with the experience it offers. 


In addition to expanding the scope of criteria to include subjective evaluations, we believe that another 
way to ensure comprehensiveness of graduate program rankings is by making sure numerous stakehold- 
ers provide input in the process. Although current students offer an important viewpoint, they cannot 
provide feedback for all relevant graduate program characteristics. An additional group of key stake- 
holders are program alumni, as these individuals are in a better position to evaluate how their programs 
prepared them for the “real world.” Accordingly, we included alumni perceptions of the program as a 


separate dimension of graduate program rankings. However, in order to reach alumni who had a good 
memory of their programs and following job search experiences, we limited our alumni sample to those 
who graduated in the last 5 years. This also allowed us to collect data from alumni who have been in the 
program relatively recently and thus are likely knowledgeable about the current state of the program 
(i.e., their perceptions would better reflect the outcome of the program in its current form). Finally, two 
separate surveys were created, one for each group of informants. The first survey, for current students 
in l-O master’s degree programs, included items assessing program culture, program resources, and sat- 
isfaction. The second survey, for recent program alumni, included items assessing whether the program 
was successful in preparing them for employment. 


Method 
Procedure 


After reviewing the existing literature on organizational culture, performance, and satisfaction measures, 
the research team developed surveys targeted at I-O programs’ current students and recent alumni. For 
each dimension, the team created a list of items based on other measures when possible (e.g., Bierer, Fish- 
leder, Dannefer, Farrow, & Hull, 2004; Kraiger & Abalos, 2004; O’Reilly, Chatman, & Caldwell, 1991). Sur- 
vey items used in the current study are in Appendix A. Email addresses for 117 master’s degree program 
directors were obtained from SIOP, and each program director was sent an email containing information 
about the project and a request to forward the appropriate survey link to current students and recent 
alumni (i.e., former students who graduated within the last 5 academic years, as of spring 2017). 


Student Survey 


Program culture. 

Student perceptions of program culture was measured by 14 items written for this study based on organi- 
zational culture profile items described in O’Reilly et al. (1991). Participants were asked to indicate their 
level of agreement regarding program cultural elements on a five-point scale ranging from 1 = strongly dis- 
agree to 5 = strongly agree. Sample items include: “I am able to achieve a balance between my work in the 
program and life outside the program” and “This program is accepting of people from various backgrounds 
and perspectives.” Cronbach’s alpha for internal consistency is .92 in the current sample. 


Program resources. 

Student perceptions of program resources were measured by 13 items adapted from Bierer et al. 
(2004). Participants were asked to indicate the extent to which their program offers access to a variety 
of resources on a five-point scale ranging from 1 = strongly disagree to 5 = strongly agree. After the 
prompt “My program offers me access to...,” the survey referenced a variety of resources including ca- 
reer development services, statistical software, and program alumni. Cronbach’s alpha for internal con- 
sistency is .88 in the current sample. 


Program satisfaction. 

Students’ satisfaction with their programs was measured by 18 items developed for this study. Partici- 
pants were asked to indicate the extent to which they are satisfied with several aspects of the program 
on a five-point scale ranging from 1 = very dissatisfied to 5 = very satisfied. The survey noted program 
characteristics such as class sizes, course offerings, and internship opportunities. Cronbach’s alpha for 
internal consistency is .93 in the current sample. 


Alumni Survey 


Alumni perceptions of the program were measured using nine items developed for this study. Partici- 
pants were asked to indicate the extent to which they agree with statements on a five-point scale rang- 
ing from 1 = strongly disagree to 5 = strongly agree. Sample items include “I feel the program has pre- 
pared me well for my career” and “I would encourage others to apply to this program.” Cronbach’s al- 
pha for internal consistency is .87 in the current sample. 


Results 


As described above, the goal of this study was to produce rankings of I-O master’s degree programs by using 
subjective evaluations of students and alumni. A total of 975 participants provided the data for the study: 
594 current students from 44 programs and 381 alumni from 36 programs. However, some programs had 
very little data on which to base the rankings. For these programs, one or two individuals with very high or 
very low evaluations could potentially skew their mean score and unduly influence their program ranking. To 
avoid this possibility, we excluded programs from the rankings if they had a sample size less than one SD be- 
low the mean for each informant group (Kraiger & Abalos, 2004). The mean sample size across programs was 
13 for students (SD = 8.4) and 11 for alumni (SD = 8). Thus, programs were included in the rankings if they 
had at least five student respondents and at least three alumni respondents, which led to the elimination of 
six programs from the student survey and 12 programs from the alumni survey. 


The rankings for student evaluations of program culture, program resources, and satisfaction with the 
program appear in Tables 1-3, respectively. We only included the 25 highest-ranked programs in the re- 
sults (see Appendix B for a complete, nonranked list of all programs from which we received any re- 
sponses). As seen in Table 1, the top three schools in program culture are Indiana University-Purdue Uni- 
versity Indianapolis (M = 4.83, SD = .15), Appalachian State University (M = 4.77, SD = .19), and Xavier 
University (M = 4.75, SD = .26). As see in Table 2, the top three schools in program resources are Appala- 
chian State University (M = 4.63, SD = .32), Minnesota State University-Mankato (M = 4.55, SD = .54), 
and Indiana University-Purdue University Indianapolis (M = 4.54, SD = .27). As seen in Table 3, the top 
three schools in student satisfaction are Appalachian State University (M = 4.69, SD = .28), Missouri 
State University (M = 4.66, SD = .27), and San Diego State University (M = 4.60, SD = .37). 


Next, we examined evaluations by program alumni. As seen in Table 4, the top-three programs in this rank- 
ing were The University of Georgia (M = 4.65, SD = .25), University of Maryland—Baltimore County (M = 
4.63, SD = .17), and Minnesota State University-Mankato (M = 4.61, SD = .44). Finally, we examined the ex- 
tent to which ratings by students on different dimensions and alumni ratings overlapped. This was done by 
creating a separate dataset of program means on each dimension (with programs being the level of analy- 
sis) and examining the correlation between means of ratings in separate dimensions. As seen in Table 5, all 
the correlations were strong and statistically significant, especially among student-rated dimensions. The 
most noteworthy correlations were correlations between alumni ratings and different dimensions of stu- 
dent evaluations. The correlations between alumni evaluations and student evaluations of program cul- 
ture, resources, and satisfaction were .49 (p < .01), .39 (p < .05), and .45 (p < .01), respectively. 


Table 1 
Program Rankings by Student Perceptions of Program Culture 
Rank Program name M SD N 
1 Indiana University-Purdue University at Indianapolis 4.83 0.15 5 


2 Appalachian State University 4.77 0.19 24 


3 Xavier University 4.74 0.26 14 
4 University of Maryland 4.71 0.25 19 
5 Missouri State University 4.66 0.30 17 
6 San Francisco State University 4.65 0.29 12 
7 The University of Tennessee at Chattanooga 4.64 0.33 22 
8 University of Hartford 4.60 0.23 8 
9 Western Kentucky University 4.58 0.42 8 
10 Minnesota State University-Mankato 4.53 0.77 18 
11 St. Mary's University 4.52 0.50 15 
12 San Diego State University 4.51 0.34 8 
13 Florida Institute of Technology 4.45 0.43 27 
14 Emporia State University 4.42 0.24 10 
15 Eastern Kentucky University 4.39 0.47 5 
16 Middle Tennessee State University 4.39 0.42 25 
17 University of Baltimore 4.38 0.61 24 
18 The University of Georgia 4.37 0.45 25 
19 University at Albany, State University of New York 4.36 0.30 5 
20 Saint Cloud State University 4.35 0.47 11 
21 Albizu University 4.31 0.70 28 
22 University of West Florida 4.30 0.56 5 
23 University of New Haven 4.26 0.45 20 
24 George Mason University 4.25 0.52 15 
25 Roosevelt University 4.24 0.29 6 
Table 2 
Program Rankings by Student Perceptions of Program Resources 

Rank Program name M SD N 
1 Appalachian State University 4.63 0.32 24 
2 Minnesota State University-Mankato 4.55 0.54 18 
3 Indiana University-Purdue University at Indianapolis 4.54 0.27 5 
4 Missouri State University 4.53 0.38 17 
5 Xavier University 4.47 0.44 14 
6 The University of Tennessee at Chattanooga 4.41 0.38 22 
7 St. Mary's University 4.31 0.52 15 
8 Western Kentucky University 4.28 0.38 8 
9 University of West Florida 4.25 0.46 5 
10 Saint Cloud State University 4.24 0.37 11 
11 Middle Tennessee State University 4.18 0.47 25 
12 Eastern Kentucky University 4.14 0.31 5 
13 San Francisco State University 4.13 0.36 12 
14 San Diego State University 4.11 0.64 8 
15 University of Maryland 4.08 0.59 19 
16 University of Hartford 4.01 0.52 8 
17 Valdosta State University 3.99 0.55 12 
18 University of Minnesota-Duluth 3.91 0.55 8 
19 University of Baltimore 3.91 0.55 24 


20 University of Central Florida 3.91 0.66 14 
21 Florida Institute of Technology 3.86 0.61 26 
22 University of Nebraska-Omaha 3.85 0.60 23 
23 Montclair State University 3.83 0.59 16 
24 Emporia State University 3.80 0.52 10 
25 The University of Texas at Arlington 3.78 0.55 12 
Table 3 
Program Rankings by Student Satisfaction 
Rank Program name M SD N 
1 Appalachian State University 4.69 0.28 24 
2 Missouri State University 4.66 0.27 17 
3 San Diego State University 4.60 0.37 8 
4 San Francisco State University 4.58 0.37 12 
5 Indiana University-Purdue University at Indianapolis 4.54 0.28 5 
6 University of Maryland 4.53 0.40 18 
7 Minnesota State University-Mankato 4.53 0.77 18 
8 The University of Tennessee at Chattanooga 4.51 0.31 22 
9 Middle Tennessee State University 4.51 0.42 25 
10 Western Kentucky University 4.43 0.37 8 
11 University of West Florida 4.38 0.25 5 
12 Eastern Kentucky University 4.37 0.34 5 
13 Xavier University 4.36 0.62 14 
14 St. Mary's University 4.35 0.46 15 
15 University of Hartford 4.32 0.47 8 
16 University of Baltimore 4.32 0.64 23 
17 Roosevelt University 4.30 0.59 5 
18 Florida Institute of Technology 4.26 0.55 26 
19 The University of Georgia 4.24 0.50 26 
20 Albizu University 4.20 0.82 28 
21 Southern Illinois University-Edwardsville 4.17 0.54 16 
22 Touro College 4.16 0.87 24 
23 Saint Cloud State University 4.13 0.50 11 
24 George Mason University 4.12 0.68 15 
25 University at Albany, State University of New York 4.10 0.55 5 
Table 4 
Program Rankings by Alumni Perceptions 
Rank Program name M SD N 
1 The University of Georgia 4.65 0.25 8 
2 University of Maryland-Baltimore County 4.63 0.17 3 
3 Minnesota State University-Mankato 4.61 0.44 8 
4 Xavier University 4.58 0.28 4 
5 Middle Tennessee State University 4.51 0.23 9 
6 The University of Tennessee at Chattanooga 4.48 0.24 7 
7 Indiana University-Purdue University at Indianapolis 4.44 0.61 19 
8 Valdosta State University 4.37 0.42 3 


9 Appalachian State University 4.36 0.36 8 


10 Albizu University 4.29 0.39 20 
11 California State University, San Bernardino 4.29 0.54 24 
12 San Francisco State University 4.23 0.37 46 
13 San Diego State University 4.22 0.29 3 
14 George Mason University 4.17 0.59 9 
15 University of Baltimore 4.17 0.56 4 
16 Western Kentucky University 4.11 0.41 10 
17 University of Central Florida 4.10 0.56 10 
18 Missouri State University 4.07 0.52 8 
19 University of West Florida 4.05 0.27 9 
20 Florida Institute of Technology 4.01 0.73 16 
21 Elmhurst College 3.99 0.45 8 
22 The University of Texas at Arlington 3.98 0.59 7 
23 University of Hartford 3.98 0.81 10 
24 University of Nebraska-Omaha 3.91 0.73 11 
25 East Carolina University 3.88 0.52 16 
Table 5 
Means, Standard Deviations, and Correlations of Study Variables 
1 2 3 4 5 
1. Student culture mean --- 
2. Student resources mean 66°" --- 
3. Student satisfaction mean 83°" 7r” --- 
4. Student overall mean 94% 83°" 1" --- 
5. Alumni satisfaction mean 49" 39° AS 51” --- 
Mean 4.32 3.98 4.21 4.16 4.10 
SD 0.29 0.33 0.32 0.29 0.31 
* p< .05, ** p< .01., *** p < .001 
Discussion 


The present report diverges from previous ranking attempts by incorporating subjective criteria such as 
culture, satisfaction, and program resources rather than relying only on research productivity outcomes. 
By doing so, the current study accomplishes two important objectives. First, it addresses issues that 
have a direct relevance to how students choose a master’s program but have received little attention in 
the extant literature. For example, assessing students’ perceptions of their program resources may provide 
an important context for whether prospective students believe they will be successful in a certain pro- 
gram. Second, it provides a more comprehensive examination of the overall effectiveness of master’s pro- 
grams by examining how programs prepare their students for their future careers based on subjective 
evaluations of program alumni. As described above, subjective evaluations of current students and alumni 
may be more important to prospective students compared to objective outcomes such as research 
productivity. Therefore, we believe these rankings are likely to be of significant value to prospective I-O 
graduate students who are engaged in the important decision of where to apply for graduate school. 


It is important to note that the top three schools as ranked by their current students do not appear in 
the top three alumni ranked programs. This demonstrates the importance of evaluating program effec- 
tiveness from the perspectives of multiple stakeholders. Current students may have differing opinions 


about their master’s programs while they are in attendance as compared to after starting their careers. 
It is likely that satisfaction with their program after graduation may be influenced by factors other than 
those that are important for student experience. Factors such as the ability to find a job, how quickly 
they are hired, how well the program prepared them for work, the kinds of jobs they were able to find, 
or getting into a PhD program after graduating from a master’s program may influence alumni percep- 
tions of their programs. Future ranking efforts should continue including this important perspective in 
evaluations of graduate programs, perhaps even considering the kinds of jobs alumni take. 


There were some limitations to this study, which largely focus around the availability of respondents and 
the use of subjective measures. Some programs have not been established long enough to produce a 
sufficient number of alumni to provide rankings. Some other programs were smaller in size and there- 
fore had fewer current students to provide rankings. Accordingly, program age and size are important 
limitations that should be acknowledged. Another limitation is that due to the subjective nature of the 
evaluations sought, some program directors may have refused to provide data. Kraiger and Abalos 
(2004) mention concerns by program directors that some criteria would put them at an inherent disad- 
vantage (e.g., program cost for schools in metropolitan areas) as one reason for some programs not be- 
ing represented in the evaluations. Even though we did not receive any feedback from program direc- 
tors with such concerns, this may be one reason for some programs not providing data to our evalua- 
tions. Regardless of reason, of 117 programs to which our efforts were directed, we received data from 
only 44 programs. However, despite these limitations, we believe the rankings reported here provide 
valuable information for potential graduate students and others interested in l-O master’s degree pro- 
gram quality. We recommend that these rankings be used in conjunction with others, including those 
developed in response to SIOP’s recent Call for Proposals (Salter et al., 2016). Together, these rankings 
provide a very comprehensive base for evaluating program success. 


References 


Beiler, A. A., Zimmerman, L. M., Doerr, A. J., & Clark, M. A. (2014). An evaluation of research productivity among l- 
O psychology doctoral programs. The Industrial-Organizational Psychologist, 51 (3), 40-52. 

Bierer, S. B., Fishleder, A. J., Dannefer, E., Farrow, N., & Hull, A. L. (2004). Psychometric properties of an instrument 
designed to measure the educational quality of graduate training programs. Evaluation & the Health Profes- 
sions, 27, 410-424. 

Gibby, R. E., Reeve, C. L., Grauer, E., Mohr, D., & Zickar, M. J. (2002). The top I-O psychology doctoral programs of 
North America. The Industrial-Organizational Psychologist, 39 (4), 17-25. 

Kraiger, K., & Abalos, A. (2004). Rankings of graduate programs in I-O psychology based on student ratings of qual- 
ity. The Industrial-Organizational Psychologist, 42 (1), 28-43. 

Oliver, J., Blair, C. A., Gorman, A., & Woehr, D. J. (2005). Research productivity of |-O psychology doctoral programs 
in North America. The Industrial-Organizational Psychologist, 43 (1), 55-63. 

O'Reilly, C. A., Chatman, J., & Caldwell, D. F. (1991). People and organizational culture: A profile comparison ap- 
proach to assessing person-organization fit. Academy of Management Journal, 34, 487-516. 

Payne, S. C., Succa, C. A., Maxey, T. D., & Bolton, K. R. (2001). Institutional representation in the SIOP conference 
program: 1986-2000. The Industrial-Organizational Psychologist, 9 (1), 53-60. 

Pritchard, R. J., Morrow, D., & Marshall, J. C. (2005). School and district culture as reflected in student voices and 
student achievement. School Effectiveness and School Improvement, 16, 153-177. 

Salter, N. P., Allen, J. A., Gabriel, A. S., Sowinski, D., & Naidoo, L. (2016). Call for proposals for I-O graduate program 
rankings. The Industrial-Organizational Psychologist, 54 (1). 

Sirota, D., Mischkind, L. A., & Meltzer, M. |. (2008). Enthusiastic employees. In R. J. Burke, C. L. Cooper, R. J. Burke, 
C. L. Cooper (Eds.), Building more effective organizations: HR Management and performance in practice (pp. 
35-56). New York, NY: Cambridge University Press. 


Tichnor-Wagner, A., Harrison, C., & Cohen-Vogel, L. (2016). Cultures of learning in effective high schools. Educa- 
tional Administration Quarterly, 52, 602-642. 

Winter, J. L., Healy, M. C., & Svyantek, D. J. (1995). North America’s top I-O psychology/doctoral programs: U.S. 
News and World Report revisited. The Industrial-Organizational Psychologist, 33 (1), 54-58. 


Xenik 
u 


ou, A., & Simosi, M. (2006). Organizational culture and transformational leadership as predictors of business 
nit performance. Journal of Managerial Psychology, 21, 566-579. 


APPENDIX A 


Student Survey-Program Culture 
Please indicate the extent to which you agree or disagree with the following statements (1 = strongl 


disagree, 5 = strongly agree) 


| am able to achieve a balance between my work in the program and life outside the program 
Students in the program are supportive of each other 

| have meaningful relationships with program faculty 

The faculty in my program care about me as a person 

Faculty are engaged in the program and its students 

Faculty in my program are motivated to provide the best environment for students’ professional 
development 

| have been treated fairly by the faculty in my program 

| am proud to be a student in this program 

Faculty take graduate student ideas seriously 

Students are invested in the success of other students 

Faculty have reasonable expectations of students 

| am given timely and constructive feedback 

There is unhealthy competition within the program 

This program is accepting of people of various backgrounds and perspectives 


Student Sample-Program Resources 


M 
e 
e 


rogram offers me access to... (1 = strongly disagree, 5 = strongly agree 
Career development services 
On-campus study spaces 
Dedicated spaces for graduate students 
Counseling services 
Statistical software 
Adequate library resources 
Mentoring 
Conferences 
Certifications and training (outside of classes) 
Funded assistantship 
Scholarships 
Funding for professional development activities (conferences, training, etc.) 
Program alumni 


Student Survey-Satisfaction With the Program 
Please indicate the extent to which you are satisfied with the following aspects of your program. (1 = 


very dissatisfied, 5 = very satisfied) 


Faculty support and accessibility 

Quality of instruction 

Balance between applied and academic emphases 
Quality of research in the program 

Connection with I-O, HR, and related communities 
Variety of course offerings 

Class size 

Culture of the program 

Availability of educational resources 

Internship and other professional opportunities 
Alumni engagement 

Engagement with the program during application process 
Student diversity 

Faculty diversity 

Student relationships 

Financial support 

Website and social media presence 

How well the program is preparing you for your career 


Alumni Survey 
Please indicate the extent to which you agree or disagree with the following statements. (1 = strong! 
disagree, 5 = strongly agree 


| like to stay updated about current events in the program 

| like to participate in available alumni events/opportunities 

| would like to donate money to the program 

The program is keeping me updated about current events/developments 

| have been provided with the necessary skills to succeed in my current career 
| like to keep in touch with faculty 

| feel the program has prepared me well for my career 

| feel the program has helped me develop my soft skills (e.g. leadership, communication, public 
speaking) 

| would encourage others to apply to this program 

| am proud to be an alumnus of this program 


APPENDIX B 
Listing of All Programs Submitting at Least One Response 


Albizu University 


Angelo State University 


Appalachian State University 


California State University, San Bernardino 


Clemson University 


Cleveland State University 


East Carolina University 


Eastern Kentucky University 


Elmhurst College 


Emporia State University 


Florida Institute of Technology 
George Mason University 


Illinois Institute of Technology 


Indiana University-Purdue University at Indianapolis 
Lamar University 


Middle Tennessee State University 


Minnesota State University-Mankato 
Missouri State University 


Montclair State University 


Roosevelt University 
Saint Cloud State University 


San Diego State University 


San Francisco State University 
Southern Illinois University- Edwardsville 


St. Mary's University 


The University of Georgia 
The University of New Haven 


The University of Tennessee at Chattanooga 


The University of Texas at Arlington 


The University of Tulsa 


Touro College 


University at Albany, State University of New York 


University of Baltimore 


University of Central Florida 


University of Hartford 


University of Maryland 


University of Maryland-Baltimore County 


University of Minnesota-Duluth 


University of Nebraska-Omaha 


University of New Haven 


University of West Florida 


Valdosta State University 


Western Kentucky University 
Xavier University 


The Interdisciplinarity of I-O Psychology PhD Programs and Faculty 


Richard N. Landers, Michael B. Armstrong, Adrian B. Helms, and Alexis N. Epps 
Old Dominion University 


In this article, we propose a new criterion for use in evaluating and ranking graduate training programs 
in l-O psychology: their “interdisciplinarity,” which we conceptually define as the degree to which they 
contribute to and influence disciplines beyond I-O psychology, defined as publications in those disci- 
plines and citations by those disciplines, respectively. We also present rankings of programs by their in- 
terdisciplinarity using various specific operationalizations and provide a listing of the secondary field of 
focus for all current individual l-O psychology PhD programs. If you want to skip the details, you can find 
an alternative way of looking at the summary rankings and tables below in the web app found at 
https://tntlab.shinyapps.io/io_programs/, which provides the same data reported here plus summaries 
of individual faculty member I-O and interdisciplinary publication productivity, with the ability to filter, 
sort, and search. 


Why Study Program Interdisciplinarity? 


A scientific discipline can be defined as a group of researchers (a) that agree upon central problems to 
be solved, (b) rely upon at least some agreed-upon facts relevant to solving those problems, and (c) pro- 
duce explanations, goals, and theories to address those problems (Wagner et al, 2011). Interdisciplinary 
research integrates information, data, techniques, tools, perspectives, concepts, and/or theories from 
two or more disciplines to advance knowledge or to solve problems whose solutions are beyond the 
scope of a single research discipline (National Academy of Sciences, National Academy of Engineering, 
and Institute of Medicine, 2005). Thus, the “interdisciplinarity” of a graduate program in I-O psychology 
is the degree to which a given program emphasizes the importance of interdisciplinary research via pub- 
lication in non-l-O outlets (i.e., interdisciplinary publication) and/or fosters citations within the research 
of non-l-O disciplines (i.e., interdisciplinary impact). 


An interdisciplinary approach to education may not be the first feature that comes to mind when con- 
sidering which graduate program to apply to or choose, but we contend that interdisciplinary education 
is important for a variety of reasons and worth consideration by graduate school applicants. Interdisci- 
plinary education and research can benefit both students’ development of valuable KSAOs as well as the 
progress within a given scientific field such as l-O psychology. 


With interdisciplinary education, students gain a comprehensive and holistic understanding of science 
and research (Spelt, Biemans, Tobi, Luning, & Mulder, 2009). With expanded and integrated knowledge 
structures, students can develop better critical thinking skills and creative problem solving (Jones, 2009; 
Spelt et al., 2009). They learn how to communicate across disciplines, building better networks of collab- 
oration (Jones, 2009; Kleinberg, 2008). A broader skill set is developed that can be applied beyond a sin- 
gle given discipline, including lifelong learning, the ability to change perspectives, and the ability to cope 
with complex issues (Jones, 2009; Spelt et al., 2009). 


Interdisciplinary research is becoming an integral feature of modern science (National Academy of Sci- 
ences, National Academy of Engineering, and Institute of Medicine, 2005). The inherent complexity of 
nature, society, and technology necessitates a broader, comprehensive knowledge base to fully under- 


stand the phenomena at play that can only be obtained through interdisciplinary work. Through inter- 
disciplinary research, scientists can build from the work of other disciplines, reducing duplication of ef- 
fort and streamlining the construction of new knowledge to address modern problems. For example, I-O 
psychology has recently begun to investigate the use of natural language processing to assess psycho- 
logical constructs (Campion, Campion, Campion, & Reider, 2016). |-Os do not need to reinvent the wheel 
here: Linguists, cognitive scientists, and computer scientists have already been working on natural lan- 
guage processing techniques that can be applied to our areas of interest, correlating language categories 
with personality, emotions, and other psychological characteristics (Tausczik & Pennebaker, 2010), as 
well as automatically scoring written essays (Deane, 2013). 


More practically, research funding organizations have recognized the shifts in nature, society, and tech- 
nology that require interdisciplinary approaches to science, with many now giving special preference to 
interdisciplinary work that cannot be accomplished with expertise from a single discipline alone. De- 
spite this shift and increasing recognition, I-O psychology and other domains have no convenient means 
of identifying the interdisciplinarity of a given academic or research-based program. The rankings pre- 
sented here address this challenge. 


How to Measure Interdisciplinarity 


Interdisciplinarity as a research topic has been conceptualized broadly to include the integrations and 
interconnections of various research perspectives, concepts, and theories by individuals, facilities, or 
even countries (Porter, Cohen, Roessner, & Perreault, 2007). The practical question that remains is how 
to measure interdisciplinarity within these contexts in such a way that interdisciplinarity of entities can 
be meaningfully compared. This idea is studied within the field of bibliometrics (Wagner et al., 2011), 
defined as the study of the pattern or model of publications with the use of statistical analysis (Narin, 
1976). Techniques such as citation analysis and evaluation of spatial distances have helped to broaden 
the perspective of interdisciplinarity by measuring aspects of scholarly productivity such as citation 
counts and the breadth of encompassed research areas. 


Perhaps the simplest type of bibliometric research, citation analysis involves study of the relationships 
between a part or the whole of a cited document and a part or the whole of the citing document (Smith, 
1981) to identify in what specific ways the authors of a work were influenced by others. It also considers 
the relationship between individual papers and the works the paper has cited (McBurney & Novak, 
2002). In contrast, a study by Nichols (2014) created a latent topic model of text contained within Na- 
tional Science Foundation’s award and proposal database to identify disciplinary influences, which re- 
vealed that disciplines that usually have less interdisciplinary interaction typically had less specialized 
language in their field; therefore, disciplines that have more interdisciplinarity are ones that have more 
general language. These approaches illustrate the varying perspectives taken to understand interdiscipli- 
narity; as a result, there is no single best-in-class approach widely accepted in this research literature. 
However, citation analysis emerged for us as the most interpretable for those without expertise in bibli- 
ometrics and thus became the focus of our rankings. 


Present Ranking Methodology 
To rank programs in l-O psychology, we took a citation analysis approach which involved nine steps: 


1. We by-hand created a list of all |-O psychology programs that currently offer PhDs as recorded in 
SIOP’s online graduate program search tool. This includes any I-O program from around the world 


that has signed up to be a part of the list, but due to SIOP’s headquarters and primary membership, 
the list is primarily made up of US programs. This was initially a list of 73. 

We by-hand searched the websites of each of those programs to determine (a) if they really were a 
traditional |-O program, and (b) who their faculty were. This was a bit of a subjective judgment, but 
ultimately, we included any program explicitly labeled “I-O psychology” and considered everything 
else to be an explicitly interdisciplinary or “l-O adjacent” program, resulting in a final list of 53 I-O 
programs, 14 adjacent, and 6 non-I-O. We considered adjacent programs to be less interesting for 
the rankings, because a graduate student applying to a program in (for example) “Applied Organiza- 
tional Sciences” should already know that the program’s going to be interdisciplinary. We still ana- 
lyzed these programs, but they are not included in the tabular rankings. Additionally, if faculty were 
listed as having a primary home outside of the I-O psychology program (most common with affili- 
ated management faculty), those faculty were not included. This was done in early spring 2017, so 
these placements are based on the database and website content at this point. 

We by-hand linked each faculty member to their id numbers in Elsevier’s Scopus database, which is a 
broad, cross-disciplinary database of journal publications and published conference proceedings from 
all academic fields. Some faculty had as many as three Scopus IDs due to variations in their names at 
time of publication and were combined by hand. Research has supported Scopus as having greater 
coverage than Web of Science and superior accuracy in comparison to Google Scholar, its primary 
competitors in bibliographic search (Falagas, Pitsouni, Malietzis & Pappas, 2007). It is generally re- 
garded as one of the most comprehensive databases of scholarly publications currently available, in- 
cluding journals, books, book chapters, and conference proceedings, but due to the complexities of 
academic publishing, its accuracy should not be considered as absolute (Mongeon & Paul-Hus, 2016). 
Using API scraping techniques (Landers, Brusso, Cavanaugh & Collmus, 2016) and an institutional 
membership to the Scopus database, we algorithmically, using R and the rscopus library, down- 
loaded a list of every publication by every faculty member in our list from Scopus. We also down- 
loaded Scopus’ categorization of each journal these publications appeared in by discipline. This re- 
sulted in a list of 15,554 publications, although publications were listed once per faculty not per 
publication. Although this introduced nonindependence into the dataset, as 2087 publications 
(13.4%) have more than one I-O faculty author, we preferred this to alternative strategies, such as 
considering only first-authored publications or developing some sort of “contribution” index. Be- 
cause authors on interdisciplinary publications are frequently not first in author order, we feared 
this would cut down our list of interdisciplinary publications dramatically and artificially. 

We algorithmically downloaded a list of every article citing any one of those articles, also collecting 
Scopus’ disciplinary categorization for these citations. This dataset was harvested in fall 2017 and 
resulted in a list of 577,120 citations, with the same caveat as described above. 

Using these data, filtered to only include journal articles and to exclude errata, we created a list of 
unique publications by I-O PhD faculty in reverse order of popularity, which appears in 1. We also 
created something that looks vaguely like a scree plot using these data, which appears in Figure 1. 
Using this plot and table, we had hoped to identify a meaningful break for contrasting “core l-0” 
with “interdisciplinary” but were unable to find a clear division point except when comparing Jour- 
nal of Applied Psychology with everything else. At the suggestion of two reviewers, we consulted 
the websites of each of the top 40 to determine the training of the editor-in-chief and, in cases of 
ambiguity, the associate editors. In other words, we looked to see if |-Os made editorial decisions 
for each of these journals. Decisions based upon this standard were clear in all cases except for the 
Journal of Vocational Behavior (JVB), which also appears ranked 4* in Table 1. At the time of our 
check, the editor-in-chief of this journal held a degree in Counseling Psychology, and only 4 of the 8 
associate editors held degrees in I-O, with one of these editors currently holding a position in a 


Table 1 


Frequency Table of I-O Journal Publication Representation 


Publication Name Count Percent Publication Name Count Percent 
Journal of Applied Psychology 722 9.93 Career Development Quarterly > 0 29 
Journal of Business and Psychology 209 2.88 Joumal'of Nonverbal Behavior 20 0. 28 
Journal of Organizational Behavior 197 2.71 Journal of Creative Behavior 19 0.26 
Journal of Vocational Behavior 195 2.68 Journal of Educational Psychology 19 0.26 
Human Performance 174 2.39 Psychological Methods 19 0.26 
Industrial and Organizational Psychology 158 2.17 Canadian Journal of Behavioural Science 18 0.25 
Journal of Occupational Health Psychology 153 2.11 Journal of Counseling Psychology 18 0.25 
International Journal of Selection and Assessment 150 2.06 Multivariate Behavioral Research 18 0.25 
Journal of Applied Social Psychology 144 1.98 Psychological Science 18 0.25 
Leadership Quarterly 141 1.94 Consulting Psychology Journal 17 0.23 
Personnel Psychology 139 1.91 Frontiers in Psychology 17 0.23 
Personality and Individual Differences 132 1.82 International Journal of Aviation Psychology 17 0.23 
Organizational Behavior and Human Decision Processes 128 1.76 Journal of Career Development 17 0.23 
Organizational Research Methods 113 1:55 Psychology of Women Quarterly 17 0.23 
Journal of Management 100 1.38 Canadian Psychology 16 0.22 
Human Resource Management Review 87 1.20 Current Directions in Psychological Science 16 0.22 
Journal of Personality and Social Psychology 84 1.16 Journal of Applied Behavior Analysis 16 0.22 
Journal of Occupational and Organizational Psychology 81 1.11 Journal of Behavioral Decision Making 16 0.22 
Educational and Psychological Measurement 17 1.06 Journal of Leadership and Organizational Studies 16 0.22 
Human Factors 66 0.91 Motivation and Emotion 16 0.22 
Military Psychology 63 0.87 Psychology and Aging 16 0.22 
Applied Psychological Measurement 60 0.83 Science and Engineering Ethics 16 0.22 
Creativity Research Journal 59 0.81 Accountability in Research 15 0.21 
Applied Psychology a 55 0.76 Applied Ergonomics 14 0.19 
Computers in Human Behavior 51 0.70 Employee Responsibilities and Rights Journal 14 0.19 
Psychological Bulletin 47 0.65 Environment and Behavior 14 0.19 
Sex Roles 46 0.63 Journal of Applied Behavioral Science 14 0.19 
Work and Stress . 46 0.63 Journal of Experimental Psychology 14 0.19 
American Psychologist 42 0.58 Journal of Occupational and Environmental Medicine 14 0.19 
Group and Organization Management 4l 0.57 Journal of Occupational Psychology 14 0.19 
Journal of Managerial Psychology 40 0.55 Journal of Psychology 14 0.19 
Journal of Research in Personality 38 0.52 Public Personnel Management 14 0.19 
Journal of Social Psychology 36 0.50 Cultural Diversity and Ethnic Minority Psychology 13 0.18 
Human Relations F 35 0.48 Journal of Personnel Psychology 13 0.18 
Organizational Behavior and Human Performance 35 0.48 Asian American Journal of Psychology 12 0.17 
Small Group Research 35 0.48 Intelligence 12 0.17 
International J ournal of Stress Management 32 0.44 Joint Commission Journal On Quality and Patient Safety 12 0.17 
Journal of Personality 31 0.43 Journal of Prevention and Intervention In The Community 12 0.17 
Psychological Reports 31 0.43 Journal of Traumatic Stress 12 0.17 
Group Dynamics 30 0.41 Behavior Research Methods 11 0.15 
Journal of Career Assessment 30 0.41 Current Psychology 1l 0.15 
J ournal of Psychology ; Interdisciplinary and Applied 30 0.41 Educational Measurement: Issues and Practice 11 0.15 
Accident Analysis and Prevention 29 0.40 Journal of Experimental Psychology: Applied 11 0.15 
Human Resource Management 29 0.40 Learning and Individual Differences 11 0.15 
Journal of Business Ethics 29 0.40 Plos One 1l 0.15 
Personality and Social Psychology Bulletin 29 0.40 Psychologist-Manager Journal ll 0.15 
Academy of Management Journal 28 0.39 Academy of Management Learning and Education 10 0.14 
J ournal of Safety Research 28 0.39 American Journal of Community Psychology 10 0.14 
Teaching of Psychology 28 0.39 ‘Assessment 10 0.14 
Basic and Applied Social Psychology 27 0.37 Ergonomics 10 0.14 
European Journal of Personality aa 27 0.37 Human Resource Development Quarterly 10 0.14 
European Journal of Work and Organizational Psychology 26 0.36 International Journal of Human Resource Management 10 0.14 
Journal of Organizational Behavior Management 26 0.36 Journal of General Psychology 10 0.14 
Journal of Personality Assessment 25 0.34 Journal of Social Behavior and Personality 10 0.14 
Psychology of Aesthetics, Creativity, and The Arts 25 0.34 Perceptual and Motor Skills 10 0.14 
Stress and Health 25 0.34 Psychosomatic Medicine 10 0.14 
Ethics and Behavior 24 0.33 Safety Science 10 0.14 
Journal of Cross-Cultural Psychology 24 0.33 Social Behavior and Personality 10 0.14 
Journal of Experimental Social Psychology 24 0.33 Social Justice Research 10 0.14 
Psychological Assessment . . 22 0.30 Note. Italicized journal names indicate “core I-O” journals. 

Bulletin of The Psychonomic Society 21 0.29 


250 - 


225 - 


200 - 


Publication Count 


Journal (in descending order by frequency) 
Figure 1. 


business school. Given (a) JVB’s associate editor training and employment lean and (b) its current mission 
statement (i.e., from its webpage: “publishes original empirical and theoretical articles that contribute novel 
insights to the fields of career choice, career development, and work adjustment across the lifespan and 
which are also valuable for applications in counseling and career development programs”), we made the de- 
cision to classify JVB, at least in its current form, as “interdisciplinary.” Journals classified as core I-O using 
this system appear in italics in Table 1. 

7. Using this categorization, we algorithmically labeled each publication and citation as either interdis- 
ciplinary or not and then tabulated these counts in several distinct ways, and these tables appear in 
Tables 2-8. If a publication had multiple I-O authors, it was counted once per author in these lists, 
and authorship order was ignored. Programs we previously categorized as “I-O adjacent” appear in 
Table 9. 

8. Next, we restructured the data to represent counts across disciplines within program and reverse- 
ordered them by count, excluding the categories “Applied Psychology” and “Psychology (all),” be- 
cause these appeared as first and second for almost every program in our list. These results appear 
in Table 10, summarizing the most common interdisciplinary fields published in by each program. 

9. Finally, we created an online app that summarizes all this information, which is available at 
https://tntlab.shinyapps.io/io_ programs 


A correlation matrix of all study variables, which appears in Table 11. We can summarize the ranking 
tables thus: 


Table 2 


Full Program Listing with All Variables 


P Name State Num. Total Interdisciplinary IO Interdisciplinary Total Pubs Interdisciplinary 10O Pubs Total Interdisciplinary I0 Interdisciplinary ; Total Citations Interdisciplinary 10 Citations 
Faculty Pubs Pubs Pubs Pubs per Total Pub perFaculty PubsperFaculty per Faculty Citations Citations Citations Citations per Citation per Faculty Citations per Faculty per Faculty 
Alliant International University, Los Angeles CA 4 27 21 6 0.78 6.75 5.25 1.50 554 509 45 0.92 138.50 127.25 11.25 
Alliant International University, San Diego CA 5 102 96 6 0.94 20.40 19.20 1.20 2093 2033 60 0.97 418.60 406.60 12.00 
Auburn University AL 6 149 122 27 0.82 24.83 20.33 4.50 3154 2768 386 0.88 525.67 461.33 64.33 
Baruch College, City University of New York NY 6 86 46 40 0.53 14.33 7.67 6.67 5002 4109 893 0.82 833.67 684.83 148.83 
Bowling Green State University OH 5 224 120 104 (0.54 44.80 24.00 20.80 7975 6243 1732 0.78 1595.00 1248.60 346.40 
Central Michigan University M 5 184 114 70 062 36.80 22.80 14.00 8346 6934 1412 0.83 1669.20 1386.80 282.40 
Clemson University sc 11 397 314 83 0.79 36.09 28.55 7.55 14251 13022 1229 0.91 1295.55 1183.82 111.73 
Colorado State University co 6 214 92 122 043 35.67 15.33 20.33 9947 7800 2147 0.78 1657.83 1300.00 357.83 
DePaul University L 5 84 51 33 06l 16.80 10.20 6.60 3817 3363 454 0.88 763.40 672.60 90.80 
Florida Institute of Technology FL 8 97 57 40 0.59 12.13 7.13 5.00 1936 1621 315 0.84 242.00 202.63 39.38 
Florida Intemational University FL 3 109 60 49 055 36.33 20.00 16.33 7548 5647 1901 0.75 2516.00 1882.33 633.67 
George Mason University VA 7 302 170 132 0.56 43.14 24.29 18.86 16558 13517 3041 0.82 2365.43 1931.00 434.43 
George Washington University DC 3 69 49 20 0.71 23.00 16.33 6.67 2075 1682 393 0.81 691.67 560.67 131.00 
Georgia Institute of Technology GA 4 131 70 6l 0.53 32.75 17.50 15.25 8568 6791 1777 0.79 2142.00 1697.75 444.25 
Griffith University MG 5 270 252 18 0.93 54.00 50.40 3.60 5933 5668 265 0.96 1186.60 1133.60 53.00 
Hofstra University NY 4 36 22 14 06l 9.00 5.50 3.50 478 431 47 0.90 119.50 107.75 11.75 
Illinois Institute of Technology L 4 98 62 36 0.63 24.50 15.50 9.00 3603 3219 384 0.89 900.75 804.75 96.00 
Kansas State University KS 2 4l 25 16 061 20.50 12.50 8.00 1200 1013 187 0.84 600.00 506.50 93.50 
Louisiana State University LA 2 5 3 2 0.60 2.50 1.50 1.00 11 10 1 091 5.50 5.00 0.50 
Louisiana Tech University LA 3 7 6 1 0.86 2.33 2.00 0.33 141 118 23 0.84 47.00 39.33 7.67 
Michigan State University M 9 512 281 231 0.55 56.89 31.22 25.67 24536 19539 4997 0.80 2726.22 2171.00 555.22 
North Carolina State University NC 5 100 63 37 063 20.00 12.60 740 2880 2514 366 0.87 576.00 502.80 73.20 
Ohio University OH 1 41 17 24 o4 41.00 17.00 24.00 2718 2210 508 0.81 2718.00 2210.00 508.00 
Old Dominion University VA 4 94 60 34 0.64 23.50 15.00 8.50 2180 1810 370 0.83 545.00 452.50 92.50 
Pennsylvania State University PA 6 203 105 98 0.52 33.83 17.50 16.33 12634 10133 2501 0.80 2105.67 1688.83 416.83 
Portland State University OR 5 172 83 89 0.48 34.40 16.60 17.80 5936 4498 1438 0.76 1187.20 899.60 287.60 
Purdue University N 5 153 107 46 070 30.60 21.40 9.20 6403 5474 929 0.85 1280.60 1094.80 185.80 
Rice University TX 5 502 349 153 0.70 100.40 69.80 30.60 29131 25322 3809 0.87 5826.20 5064.40 761.80 
Roosevelt University 4 43 38 5 0.88 10.75 9.50 1.25 536 520 16 0.97 134.00 130.00 4.00 
Saint Louis University MO 4 49 36 13 0.73 12.25 9.00 3.25 1715 1602 113 0.93 428.75 400.50 28.25 
Saint Mary's University MN 7 220 161 59 0.733 31.43 23.00 8.43 9034 7853 1181 0.87 1290.57 1121.86 168.71 
Seattle Pacific University WA 4 27 23 4 0.85 6.75 5.75 1.00 502 414 88 0.82 125.50 103.50 22.00 
Texas A&M University TX 5 166 83 83 0.50 33.20 16.60 16.60 6412 5117 1295 0.80 1282.40 1023.40 259.00 
University at Albany, SUNY NY 5 78 38 40 049 15.60 7.60 8.00 1624 1358 266 0.84 324.80 271.60 53.20 
University of Akron OH 4 162 84 78 0.52 40.50 21.00 19.50 5734 4509 1225 0.79 1433.50 1127.25 306.25 
University of Calgary AB 6 224 189 35 084 37.33 31.50 5.83 8199 7333 866 0.89 1366.50 1222.17 144.33 
University of Central Florida FL 6 76 $3 23 0.70 12.67 8.83 3.83 1284 1168 116 0.91 214.00 194.67 19.33 
University of Connecticut CT 4 141 97 44 069 35.25 24.25 11.00 4301 3532 769 0.82 1075.25 883.00 192.25 
University of Georgia GA 10 419 325 94 078 41.90 32.50 9.40 21253 19401 1852 0.91 2125.30 1940.10 185.20 
University of Guelph ON 7 78 50 28 0.64 11.14 7.14 4.00 4066 3550 516 0.87 580.86 507.14 73.71 
University of Houston TX 6 202 121 81 0.60 33.67 20.17 13.50 17691 15015 2676 0.85 2948.50 2502.50 446.00 
University of Illinois at Urbana-Champaign IL 3 232 121 111 0.52 77.33 40.33 37.00 13453 10903 2550 0.81 4484.33 3634.33 850.00 
University of Minnesota MN 5 304 116 188 0.38 60.80 23.20 37.60 17070 12295 4775 0.72 3414.00 2459.00 955.00 
University of Missouri-St Louis MO 7 169 135 34 0.80 24.14 19.29 4.86 3651 3269 382 0.90 521.57 467.00 54.57 
University of Nebraska at Omaha NE 5 145 129 16 0.89 29.00 25.80 3.20 3324 3123 201 0.94 664.80 624.60 40.20 
University of Oklahoma OK 4 380 270 110 0.71 95.00 67.50 27.50 12730 10260 2470 0.81 3182.50 2565.00 617.50 
University of South Florida FL 8 488 295 193 0.60 61.00 36.88 24.13 33899 27849 6050 0.82 4237.38 3481.13 756.25 
University of Texas at Arlington T™ 3 95 87 8 0.92 31.67 29.00 2.67 3444 3312 132 0.96 1148.00 1104.00 44.00 
University of Tulsa OK 5 65 39 26 0.60 13.00 7.80 5.20 1788 1286 502 0.72 357.60 257.20 100.40 
University of Waterloo ON 6 146 88 58 0.60 24.33 14.67 9.67 6049 4965 1084 0.82 1008.17 827.50 180.67 
University of Western Ontario ON 4 185 117 68 0.63 46.25 29.25 17.00 29577 26025 3552 0.88 7394.25 6506.25 888.00 
Virginia Tech VA 3 215 184 31 0.86 71.67 61.33 10.33 4795 4172 623 0.87 1598.33 1390.67 207.67 
Wayne State University MI 5 102 62 40 0.61 20.40 12.40 8.00 4753 4103 650 0.86 950.60 820.60 130.00 


Table 3 


Programs Ranked by Raw Interdisciplinary Publication Counts 


Interdisciplinary _Interdisciplinary 


Rank Program Name IOPubs Pubs Pubs per Faculty 
1 Rice University 153 349 69.80 
2 University of Georgia 94 325 32.50 
3 Clemson University 83 314 28.55 
4 University of South Florida 193 295 36.88 
5 Michigan State University 231 281 31.22 
6 University of Oklahoma 110 270 67.50 
7 Griffith University 18 252 50.40 
8 University of Calgary 35 189 31.50 
9 Virginia Tech 31 184 61.33 
10 George Mason University 132 170 24.29 
11 Saint Mary's University 59 161 23.00 
12 University of Missouri-St Louis 34 135 19.29 
13 University of Nebraska at Omaha 16 129 25.80 
14 Auburn University 27 122 20.33 
15 University of Houston 81 121 20.17 
15 University of Illinois at Urbana-Champaign 111 121 40.33 
17 Bowling Green State University 104 120 24.00 
18 University of Western Ontario 68 117 29.25 
19 University of Minnesota 188 116 23.20 
20 Central Michigan University 70 114 22.80 
21 Purdue University 46 107 21.40 
22 Pennsylvania State University 98 105 17.50 
23 University of Connecticut 44 97 24.25 
24 Alliant International University, San Diego 6 96 19.20 
25 Colorado State University 122 92 15.33 
26 University of Waterloo 58 88 14.67 
27 University of Texas at Arlington 8 87 29.00 
28 University of Akron 78 84 21.00 
29 Portland State University 89 83 16.60 
29 Texas A&M University 83 83 16.60 
31 Georgia Institute of Technology 61 70 17.50 
32 North Carolina State University 37 63 12.60 
33 Illinois Institute of Technology 36 62 15.50 
33 Wayne State University 40 62 12.40 
35 Florida International University 49 60 20.00 
35 Old Dominion University 34 60 15.00 
37 Florida Institute of Technology 40 57 7.13 
38 University of Central Florida 23 53 8.83 
39 DePaul University 33 51 10.20 
40 University of Guelph 28 50 7.14 
41 George Washington University 20 49 16.33 
42 Baruch College, City University of New York 40 46 7.67 
43 University of Tulsa 26 39 7.80 
44 Roosevelt University 5 38 9.50 
44 University at Albany, SUNY 40 38 7.60 
46 Saint Louis University 13 36 9.00 
47 Kansas State University 16 25 12.50 
48 Seattle Pacific University 4 23 5.75 
49 Hofstra University 14 22 5.50 
50 Alliant International University, Los Angeles 6 21 5.25 
51 Ohio University 24 17 17.00 
52 Louisiana Tech University 1 6 2.00 


53 Louisiana State University 2 3 1.50 


Table 4 


Programs Ranked by Proportion of Interdisciplinary Publications per Publication 
Interdisciplinary Interdisciplinary Interdisciplinary 


Rank _ Program Name IO Pubs _ Pubs Pubs per Total Pub Pubs per Faculty 
1 Alliant International University, San Diego 6 96 0.94 19.20 
2 Griffith University 18 252 0.93 50.40 
3 University of Texas at Arlington 8 87 0.92 29.00 
4 University of Nebraska at Omaha 16 129 0.89 25.80 
5 Roosevelt University 5 38 0.88 9.50 
6 Louisiana Tech University 1 6 0.86 2.00 
7 Virginia Tech 31 184 0.86 61.33 
8 Seattle Pacific University 4 23 0.85 5.75 
9 University of Calgary 35 189 0.84 31.50 
10 Auburn University 27 122 0.82 20.33 
11 University of Missouri-St Louis 34 135 0.80 19.29 
12 Clemson University 83 314 0.79 28.55 
13 Alliant International University, Los Angeles 6 21 0.78 5.25 
14 University of Georgia 94 325 0.78 32.50 
15 Saint Louis University 13 36 0.73 9.00 
16 Saint Mary's University 59 161 0.73 23.00 
17 University of Oklahoma 110 270 0.71 67.50 
18 George Washington University 20 49 0.71 16.33 
19 Purdue University 46 107 0.70 21.40 
20 University of Central Florida 23 53 0.70 8.83 
21 Rice University 153 349 0.70 69.80 
22 University of Connecticut 44 97 0.69 24.25 
23 University of Guelph 28 50 0.64 7.14 
24 Old Dominion University 34 60 0.64 15.00 
25 Illinois Institute of Technology 36 62 0.63 15.50 
26 University of Western Ontario 68 117 0.63 29.25 
27 North Carolina State University 37 63 0.63 12.60 
28 Central Michigan University 70 114 0.62 22.80 
29 Hofstra University 14 22 0.61 5.50 
30 Kansas State University 16 25 0.61 12.50 
31 Wayne State University 40 62 0.61 12.40 
32 DePaul University 33 51 0.61 10.20 
33 University of South Florida 193 295 0.60 36.88 
34 University of Waterloo 58 88 0.60 14.67 
35 Louisiana State University 2 3 0.60 1.50 
35 University of Tulsa 26 39 0.60 7.80 
37 University of Houston 81 121 0.60 20.17 
38 Florida Institute of Technology 40 57 0.59 7.13 
39 George Mason University 132 170 0.56 24.29 
40 Florida International University 49 60 0.55 20.00 
41 Michigan State University 231 281 0.55 31.22 
42 Bowling Green State University 104 120 0.54 24.00 
43 Baruch College, City University of New York 40 46 0.53 7.67 
44 Georgia Institute of Technology 6l 70 0.53 17.50 
45 University of Illinois at Urbana-Champaign 111 121 0.52 40.33 
46 University of Akron 78 84 0.52 21.00 
47 Pennsylvania State University 98 105 0.52 17.50 
48 Texas A&M University 83 83 0.50 16.60 
49 University at Albany, SUNY 40 38 0.49 7.60 
50 Portland State University 89 83 0.48 16.60 
51 Colorado State University 122 92 0.43 15.33 
52 Ohio University 24 17 0.41 17.00 


53 University of Minnesota 188 116 0.38 23.20 


Table 5 
Programs Ranked by Raw IO Publication Counts 


IO Pubs per 

Rank _ Program Name IO Pubs _ Faculty 
1 Michigan State University 231 25.67 
2 University of South Florida 193 24.13 
3 University of Minnesota 188 37.60 
4 Rice University 153 30.60 
5 George Mason University 132 18.86 
6 Colorado State University 122 20.33 
7 University of Illinois at Urbana-Champaign 111 37.00 
8 University of Oklahoma 110 27.50 
9 Bowling Green State University 104 20.80 
10 Pennsylvania State University 98 16.33 
11 University of Georgia 94 9.40 
12 Portland State University 89 17.80 
13 Clemson University 83 7.55 
13 Texas A&M University 83 16.60 
15 University of Houston 81 13.50 
16 University of Akron 78 19.50 
17 Central Michigan University 70 14.00 
18 University of Western Ontario 68 17.00 
19 Georgia Institute of Technology 61 15.25 
20 Saint Mary's University 59 8.43 
21 University of Waterloo 58 9.67 
22 Florida International University 49 16.33 
23 Purdue University 46 9.20 
24 University of Connecticut 44 11.00 
25 Baruch College, City University of New York 40 6.67 
25 Florida Institute of Technology 40 5.00 
25 University at Albany, SUNY 40 8.00 
25 Wayne State University 40 8.00 
29 North Carolina State University 37 7.40 
30 Illinois Institute of Technology 36 9.00 
31 University of Calgary 35 5.83 
32 Old Dominion University 34 8.50 
32 University of Missouri-St Louis 34 4.86 
34 DePaul University 33 6.60 
35 Virginia Tech 31 10.33 
36 University of Guelph 28 4.00 
37 Auburn University 27 4.50 
38 University of Tulsa 26 5.20 
39 Ohio University 24 24.00 
40 University of Central Florida 23 3.83 
41 George Washington University 20 6.67 
42 Griffith University 18 3.60 
43 Kansas State University 16 8.00 
43 University of Nebraska at Omaha 16 3.20 
45 Hofstra University 14 3.50 
46 Saint Louis University 13 3.25 
47 University of Texas at Arlington 8 2.67 
48 Alliant International University, Los Angeles 6 1.50 
48 Alliant International University, San Diego 6 1.20 
50 Roosevelt University 5 1.25 
51 Seattle Pacific University 4 1.00 
52 Louisiana State University 2 1.00 
53 Louisiana Tech University 1 0.33 


Table 6 


Programs Ranked by Raw Interdisciplinary Citation Count 
Interdisciplinary _Interdisciplinary 


Rank _ Program Name IO Citations Citations Citations per Faculty 
1 University of South Florida 6050 27849 3481.13 
2 University of Western Ontario 3552 26025 6506.25 
3 Rice University 3809 25322 5064.40 
4 Michigan State University 4997 19539 2171.00 
5 University of Georgia 1852 19401 1940.10 
6 University of Houston 2676 15015 2502.50 
7 George Mason University 3041 13517 1931.00 
8 Clemson University 1229 13022 1183.82 
9 University of Minnesota 4775 12295 2459.00 
10 University of Illinois at Urbana-Champaign 2550 10903 3634.33 
11 University of Oklahoma 2470 10260 2565.00 
12 Pennsylvania State University 2501 10133 1688.83 
13 Saint Mary's University 1181 7853 1121.86 
14 Colorado State University 2147 7800 1300.00 
15 University of Calgary 866 7333 1222.17 
16 Central Michigan University 1412 6934 1386.80 
17 Georgia Institute of Technology 1777 6791 1697.75 
18 Bowling Green State University 1732 6243 1248.60 
19 Griffith University 265 5668 1133.60 
20 Florida International University 1901 5647 1882.33 
21 Purdue University 929 5474 1094.80 
22 Texas A&M University 1295 5117 1023.40 
23 University of Waterloo 1084 4965 827.50 
24 University of Akron 1225 4509 1127.25 
25 Portland State University 1438 4498 899.60 
26 Virginia Tech 623 4172 1390.67 
27 Baruch College, City University of New York 893 4109 684.83 
28 Wayne State University 650 4103 820.60 
29 University of Guelph 516 3550 507.14 
30 University of Connecticut 769 3532 883.00 
31 DePaul University 454 3363 672.60 
32 University of Texas at Arlington 132 3312 1104.00 
33 University of Missouri-St Louis 382 3269 467.00 
34 Illinois Institute of Technology 384 3219 804.75 
35 University of Nebraska at Omaha 201 3123 624.60 
36 Auburn University 386 2768 461.33 
37 North Carolina State University 366 2514 502.80 
38 Ohio University 508 2210 2210.00 
39 Alliant International University, San Diego 60 2033 406.60 
40 Old Dominion University 370 1810 452.50 
41 George Washington University 393 1682 560.67 
42 Florida Institute of Technology 315 1621 202.63 
43 Saint Louis University 113 1602 400.50 
44 University at Albany, SUNY 266 1358 271.60 
45 University of Tulsa 502 1286 257.20 
46 University of Central Florida 116 1168 194.67 
47 Kansas State University 187 1013 506.50 
48 Roosevelt University 16 520 130.00 
49 Alliant International University, Los Angeles 45 509 127.25 
50 Hofstra University 47 431 107.75 
51 Seattle Pacific University 88 414 103.50 
52 Louisiana Tech University 23 118 39.33 


53 Louisiana State University 1 10 5.00 


Table 7 


Programs Ranked by Proportion of Interdisciplinary Citations per Citation (Interdisciplinary Impact Rate) 


Interdisciplinary 

IO Interdisciplinary Citations per Interdisciplinary 
Rank Program Name Citations Citations Citation Citations per Faculty 
1 Alliant International University, San Diego 60 2033 0.97 406.60 
2 Roosevelt University 16 520 0.97 130.00 
3 University of Texas at Arlington 132 3312 0.96 1104.00 
4 Griffith University 265 5668 0.96 1133.60 
5 University of Nebraska at Omaha 201 3123 0.94 624.60 
6 Saint Louis University 113 1602 0.93 400.50 
7 Alliant International University, Los Angeles 45 509 0.92 127.25 
8 Clemson University 1229 13022 0.91 1183.82 
9 University of Georgia 1852 19401 0.91 1940.10 
10 University of Central Florida 116 1168 0.91 194.67 
11 Louisiana State University 1 10 0.91 5.00 
12 Hofstra University 47 431 0.90 107.75 
13 University of Missouri-St Louis 382 3269 0.90 467.00 
14 University of Calgary 866 7333 0.89 1222.17 
15 Illinois Institute of Technology 384 3219 0.89 804.75 
16 DePaul University 454 3363 0.88 672.60 
17 University of Western Ontario 3552 26025 0.88 6506.25 
18 Auburn University 386 2768 0.88 461.33 
19 University of Guelph 516 3550 0.87 507.14 
20 North Carolina State University 366 2514 0.87 502.80 
21 Virginia Tech 623 4172 0.87 1390.67 
22 Saint Mary's University 1181 7853 0.87 1121.86 
23 Rice University 3809 25322 0.87 5064.40 
24 Wayne State University 650 4103 0.86 820.60 
25 Purdue University 929 5474 0.85 1094.80 
26 University of Houston 2676 15015 0.85 2502.50 
27 Kansas State University 187 1013 0.84 506.50 
28 Florida Institute of Technology 315 1621 0.84 202.63 
29 Louisiana Tech University 23 118 0.84 39.33 
30 University at Albany, SUNY 266 1358 0.84 271.60 
31 Central Michigan University 1412 6934 0.83 1386.80 
32 Old Dominion University 370 1810 0.83 452.50 
33 Seattle Pacific University 88 414 0.82 103.50 
34 University of South Florida 6050 27849 0.82 3481.13 
35 Baruch College, City University of New York 893 4109 0.82 684.83 
36 University of Connecticut 769 3532 0.82 883.00 
37 University of Waterloo 1084 4965 0.82 827.50 
38 George Mason University 3041 13517 0.82 1931.00 
39 Ohio University 508 2210 0.81 2210.00 
40 George Washington University 393 1682 0.81 560.67 
41 University of Illinois at Urbana-Champaign 2550 10903 0.81 3634.33 
42 University of Oklahoma 2470 10260 0.81 2565.00 
43 Pennsylvania State University 2501 10133 0.80 1688.83 
44 Texas A&M University 1295 5117 0.80 1023.40 
45 Michigan State University 4997 19539 0.80 2171.00 
46 Georgia Institute of Technology 1777 6791 0.79 1697.75 
47 University of Akron 1225 4509 0.79 1127.25 
48 Colorado State University 2147 7800 0.78 1300.00 
49 Bowling Green State University 1732 6243 0.78 1248.60 
50 Portland State University 1438 4498 0.76 899.60 
51 Florida International University 1901 5647 0.75 1882.33 
52 University of Minnesota 4775 12295 0.72 2459.00 
53 University of Tulsa 502 1286 0.72 257.20 


Table 8 


Programs Ranked by Raw I-O Citation Count 


Interdisciplinary 
IO Interdisciplinary Citations per 
Rank Program Name Citations Citations Faculty 
1 University of South Florida 6050 27849 3481.13 
2 Michigan State University 4997 19539 2171.00 
3 University of Minnesota 4775 12295 2459.00 
4 Rice University 3809 25322 5064.40 
5 University of Western Ontario 3552 26025 6506.25 
6 George Mason University 3041 13517 1931.00 
7 University of Houston 2676 15015 2502.50 
8 University of Illinois at Urbana-Champaign 2550 10903 3634.33 
9 Pennsylvania State University 2501 10133 1688.83 
10 University of Oklahoma 2470 10260 2565.00 
11 Colorado State University 2147 7800 1300.00 
12 Florida International University 1901 5647 1882.33 
13 University of Georgia 1852 19401 1940.10 
14 Georgia Institute of Technology 1777 6791 1697.75 
15 Bowling Green State University 1732 6243 1248.60 
16 Portland State University 1438 4498 899.60 
17 Central Michigan University 1412 6934 1386.80 
18 Texas A&M University 1295 5117 1023.40 
19 Clemson University 1229 13022 1183.82 
20 University of Akron 1225 4509 1127.25 
21 Saint Mary's University 1181 7853 1121.86 
22 University of Waterloo 1084 4965 827.50 
23 Purdue University 929 5474 1094.80 
24 Baruch College, City University of New York 893 4109 684.83 
25 University of Calgary 866 7333 1222.17 
26 University of Connecticut 769 3532 883.00 
27 Wayne State University 650 4103 820.60 
28 Virginia Tech 623 4172 1390.67 
29 University of Guelph 516 3550 507.14 
30 Ohio University 508 2210 2210.00 
31 University of Tulsa 502 1286 257.20 
32 DePaul University 454 3363 672.60 
33 George Washington University 393 1682 560.67 
34 Auburn University 386 2768 461.33 
35 Illinois Institute of Technology 384 3219 804.75 
36 University of Missouri-St Louis 382 3269 467.00 
37 Old Dominion University 370 1810 452.50 
38 North Carolina State University 366 2514 502.80 
39 Florida Institute of Technology 315 1621 202.63 
40 University at Albany, SUNY 266 1358 271.60 
41 Griffith University 265 5668 1133.60 
42 University of Nebraska at Omaha 201 3123 624.60 
43 Kansas State University 187 1013 506.50 
44 University of Texas at Arlington 132 3312 1104.00 
45 University of Central Florida 116 1168 194.67 
46 Saint Louis University 113 1602 400.50 
47 Seattle Pacific University 88 414 103.50 
48 Alliant International University, San Diego 60 2033 406.60 
49 Hofstra University 47 431 107.75 
50 Alliant International University, Los Angeles 45 509 127.25 
51 Louisiana Tech University 23 118 39.33 
52 Roosevelt University 16 520 130.00 
53 Louisiana State University 1 10 5.00 


Table 9 
Full Listing of Explicitly Interdisciplinary and Non-Traditional SIOP I-O Programs with All Variables 


10 
Interdisciplinary Total Interdisciplinary Pubs Interdisciplinary Total Interdisciplinary 

Num. Total Inter. IO PubsperTotal Pubsper Pubs per per Total Interdisciplinary IO Citations per Citationsper Citation per 10 Citation 
Program Name State Program Type Faculty Pubs Pubs Pubs Pub Faculty Faculty Faculty Citations _ Citations Citation Cite Faculty Faculty per Faculty 
Alliant Int'l University, San Francisco CA Org Development 3 is 15 0 1.00 5.00 5.00 0.00 121 118 3 0.98 40.33 39.33 1.00 
Claremont Graduate University CA Organizational Behavior/Org Psychology 7 362 343 19 095 $1.71 49.00 2.71 27172 26109 1063 0.96 3881.71 3729.86 151.86 
East Carolina University NC Health Psychology 5 7% 7% 6 092 15.20 14.00 1.20 1014 922 92 0.91 202.80 184.40 18.40 
Indiana University - Purdue University Indianapolis IN Social/Org Psychology 4 87 74 13 085 21.75 18.50 3.25 2425 2154 271 0.89 606.25 538.50 67.75 
Northem Illinois University IL Social/I-O Psychology 6 281 250 31 0.89 46.83 41.67 $5.17 7237 6883 354 0.95 1206.17 1147.17 59.00 
Singapore Management University (SMU) Singapore General Psychology 6 113 86 27 0.76 18.83 14.33 4.50 5462 4274 1188 0.78 910.33 712,33 198,00 
Southem Illinois University Carbondale IL General Psychology 1 2 2 o0 1.00 22.00 22.00 0.00 622 614 8 0.99 622.00 614.00 8.00 
Teachers College, Columbia University NY Social/Org Psychology 10 216 183 33 085 21.60 18.30 3.30 6031 5360 671 0.89 603.10 536.00 67.10 
Temple University PA Social/Org Psychology 2 80 #77 3 0.96 40.00 38.50 1.50 2564 2528 36 0.99 1282.00 1264.00 18.00 
University of Cincinnati OH Experimental Psychology 4 39 29 10 074 9.75 7.25 2.50 1041 925 116 0.89 260.25 231.25 29.00 
University of Maryland MD Social/I-O Psychology 7 461 400 61 0.87 65.86 $7.14 8.71 25627 24013 1614 0.94 3661.00 3430.43 230.57 
University of North Carolina Charlotte NC Organizational Sciences 9 226 154 72 068 25.11 17.11 8.00 8133 7348 785 0.90 903.67 816.44 87.22 
Washington State University WA Experimental Psychology 1 6 45 18 O71 63.00 45.00 18.00 1876 1616 260 0.86 1876.00 1616.00 260.00 
Wester Michigan Universi MI Behavior ement/I-O Psychol 5 170 170 0 1.00 34.00 34.00 0.00 2173 2149 24 0.99 434.60 429.80 4.80 


Table 10 


Jap Five rbienten Dac lo per irn Exton “Psychology fail)” and “Applied Poctology 


Program Nanse ind 4 Sih 
Alliam Inri University, San Francisco ‘Organizational Behavior and Human Resource Social Sciences (all) Business, Management and Accounting (all) Philosophy Education 
Alliant International University, Los Angeles Social Psychology Psy chology (miscellaneous) Sociology and Pobtical Science Management of Technology and Innovation 
Alliant International University, San Diego Nursing (all) Cardiology and Cardiovascular Medicine Medicine (all) Developmental and Educational Psychology 
‘Auburn University Public Health, Emwonmental and Expenmental and Cognitive Psychology Psychiatry and Mental Health Busaness and Laersatonal 
Occupational Health 
Baruch College, City University of New York Social Psychology Bises and Intermatnal Management Education Organizational Bebavior and Human Resource Decision Sciences (all) 
Bowting Green State University Social Psychology (Orpanizaticeal Behavior and Human Resource Business and Intemational Management Edscatica Decision Sciences (all) 
Central Michigan University Arts and Humanities (miscellaneous) Medicane (all) Organizational Behavior sed Human Resource Education Social Psychology 
Claremont Graduate University Social Psychology Busaness and International Management ‘Sociology and Political Science (Communication Economics and Econometrics 
Cheusson University Social Psychology Psychastry and Memtal Health Public Health, Environmental and Clinical Psychology Physiology 
Health 
Colorado State University Organizational Behavior and Human Resource Business and International Management Business, Management and Accounting (all) Education Social Psychology 
DePaul University Social Psychology Business and International Management Decision Sciences (all) Gender Studies Management of Technology and Innovation 
East Carolina University Education Social Psychology Clinical Psychology Developmental and Educational Psychology Psychiatry and Mental Health 
Florida Institute of Technology Social Psychology Business and international Management ‘Business, Management and Accounting (all) Human Factors and Ergonomics Organizatiosal Behavior and Human Resource 
Florida International University Social Psychology Management of Technology and innovation Orgaeazational Behavior aed Human Resource Business, Management and Accounting (all) Gender Studies 
George Masca University Business and International Management Social Psychology Masagemest of Technology and Inaovation Organizational Behavior and Human Resource Education 
Goorge Washingtoe University Business and Intemational Management Education Social Psychology Developencetal and Educational Psychology Asts and Humanities (miscellanooss) 
Goorgia instinate of Techaology Social Psychology Hines and laterana] Masigecocnt Education Experimental and Cognitive Psychology Organizational Behavior and Human Resource 
tioaal Health Management 
Hofstra’ i! Devekymnental and Educatxmal Psychology  Medicme (all) Business and Internaticeal Management Education Social 
illinois Instinse of Technology Business and International Psychiatry and Mental Heath Social Psychology: Management of Technology and Innovation Organizatioeal Behavior and Human Resource 
Indiana University - Purdue University Social Psychology Medicane (all) ‘Nursing (all) Arts and Humanities (miscellaneous) ‘Business and International Management 
Kansas State University Developmental and Educational Psychology Education Social Psychology Business and International Management larriari eea 
Louisiana State University Decisson Sciences (all) Social Psychology NA NA NA 
Louisiana Tech University Arts and Hemmanities (miscellaneous) Commumcabon Education Health Informatics ‘Language and Linguistics 
Michigan State University Social Psychology Business and Intermabonal Management Edveabon Management of Technology and Innovate Organizaticeal Behavior and Human Resource 
North Carolina State University Decasion Scieaoes (all) Social Psychology Business and Internatioeal Management Arts sod Hemsanities (miscellancous) 
Northem Illinois University Social Education Clinical Sociology and Political Science Developmental and Edscational Paychotogy 
Ohio University Fince Management of Technology and lamovation Medicine (all) Social Psychology 
Old Dominioe University Business and International Management Computer Science Applacations Social Psychology Medicine (all) Public Health, Environmental and 
Occupational Health 
Pennsylvania State University Business and International Management Education Social Psychology: Management of Teckmology and Innovate SS Behavior and Human Resource 
Portland State University Social Psychology Business and International Management Public Health, Enviromental and Management of Technology and Innovation Finance 
Health 
Purdue University ‘Social Psychology Biases and [ntersatxmal Management Developmental and Educational Psychology ‘Strategy and Managesocat Decision Sexences (all) 
Rice University Social Psychology Human Factors and Ergcecenics Educatsoa Behavioral Neuroscience Experienental and Cognitive Psychology 
6 

Saint Louis University Social Psychology Bunaness and International Management Seca Poychology Education 
Saint Mary's University Public Health, Enviromental and Medicane (all) Business, Management and Accounting Social Psychology ‘Business and International Management 

5 Occupational Health N (miscellanecers) a 
Seattle Pacific University Religious Stadies Business and Intermatonal Management Organizational Behavior and Human Resource Social Psychology Developmental and Educational Psychology 

Management University (SMU) Social Psychology Business and International Management Social Sciences (all) Decision Sciences (all) Experimental and Cognitive Psychology 
Southern Illinois University Carbondale Social Psychology Education Communication Pathology and Foressie Medicine Sociology and Political Science 
Teachers College, Columbia University Social Psychology Education Organizational Behavior and Himan Resource Arts and Humanities (miscellaneous) ‘Management of Technology and Innovation 
Temple University Social Psychology Management of Technology and Innovation Developmental and Educational Psychology Arts and Humanities (miscellaneous) Business and International Management 
Texas A&M University Social Psychology Business and International Management Organizational Behavior sed Hussan Resource Education Safety, Risk, Reliability and Quality 
University at Albany, SUNY Social Psychology Busses and International Management Management of Technology and Inmovation Organizational Behavice and Human Resource Education 
University of Calgary ‘Social Psychology Binas and Intersatxmal Managecocot Pathology and Forensic Medicine Edscation Organizaticeal Behavior and Human Resource 
University of Central Florida Social Psychology Busses and [meratcna] Management Developmental and Educational Psychology Management of Tectmology and Innovation peeun Behavior and Human Resource 
University of Cincinnati Busines and International Management Education ‘Masaigement of Technology and Inavation Arty sad Hemansties (miscellaneous) Busting and Comtrestion 
University of Connecticut oa Human Factors and Ergcecenics Organizational Behavior and Human Resource Gender Studies Social Psychology 
University of Georgia Social Psychology Education Clinical Psychology Business and International Management Organizaticeal Behavior and Human Resource 
University of Guelph Business and International Management Management of Technology and Innovation Social Psychology ‘Medicine (alll) Sociology and Political Science 
Univeniity of Houston Social Psychology Education Busness and Intemational Manapement Arts sod Hummanstes (miscellaneous) Sociology and Political Science 
University of [knots at Urbana-Champaign: Social Scxences (miscellaneous) Education Social Psychology: Docaxa Scieaces (all) Experimental and Cognitive Psychology 
University of Marytand Social Psychology Busses and International Sociology and Political Sonce Developmental and Educational Psychology Arts and Humanities (miscellaneoms) 
University of Minnesota Edecation Business, Management and Accoemting (all) Social Psychology Business and International Management Management of Technology and Innovation 
University of Missouri-St Louis Social Education ‘Business and International Management Health (social science) ‘Human Factors and Ergonomics 
University of Nebraska at Omaha Developmental and Educational Psychology Social Psychology Education Business and International Management Business, Management and Accounting 

(miscellaneous) 

University of North Carolina Charlotte Business and Intemational Management Social Business, Management and Accountang (all) Arts and Humanities (miscellaneous) Education 
University of Oklaboma ‘Business and International Management Developmental and Educational Psychology Social Education Health (social science) 
University of South Florida Edscaticn Social Psychology ‘Medbcine (all) Sovial Scrences (immcellancces) Asts and Humanitees (aiscellaneces) 
University of Texas at Arliagton Social Psychology Education Developmental and Educational Psychology Catalysis Sociology and Political Science 
University of Tulsa Social Psychology Education Business and International Management Pathology and Forensic Medicine Arts and Humanities (miscellaneces) 
University of Waterloo Social Psychology Business and International Management Sociology and Political Science Education Anthcopology 
University of Westem Outario Business and International Management Education Leadership and Social Psychology Expensnental and Cognitive Psychology 
Virginia Tech Social Psychology ‘Safety, Risk, Reliatality and Quality Business and International Management Environmental Scieace (all) 
‘Washington State University Business and International Management Safety, Risk, Reliability and Quality Clinical Human Factors and Ergonomics Social Psychology 
Wayne State University Business and International Management Social Psychology Education Experimental and Cognitive Psychology Neuropsychology and Physiological 
Westem Michigan University Education Experimental and Cognitive Psychology Management of Technology and lamovation  Developenestal and Educational Psychology 
_ Wright State University Business and International Management Business, Management and Accounting (all) Social Psychology Experimental and Cognitive Psychology 


Table 11 


Means, Standard Deviations, and Correlations at Program Level of Analysis 


M SD 1 2 3 4 5 6 7 8 9 10 11 12 13 14 
I. Number of Faculty 5.06 1.91 
2. Total Publication 166.42 126.46 .60** 
3. Interdisciplinary Publications 108.64 87.06 .60** 95% 
4. IO Publications 57.77 51.71 46** .85** .64** 
5. Interdisciplinary Pubs per Total Pub 0.66 0.14 -04 -.06 .20 -.49** 
6. Total Pubs per Faculty 32.35 21.31 .07 .80** lone 69** -.12 
7. Interdisciplinary Pubs per Faculty 20.97 14.97 09 sore 81** AT** 21 .93** 
8. IO Pubs per Faculty 11.38 9.22 01 .63** 42** .84** -.62** .80** :53** 
9. Total Citations 7669.66 8028.21 AS** 86** -75** .83** -.22 -70** 58** 67** 
10. Interdisciplinary Citations 6432.68 6765.38 .46** .85** s177% J98* -.17 .69** 59** .63** 1.00** 
11. IO Citations 1236.98 1392.54 235° .80** .60** 94** -.43** 69** A48** 81** .92** .89** 
12. Interdisciplinary Citations per Citation 0.85 0.06 .06 -.15 .08 -.50** N e id -.25 03 -.61** -.20 -.14 -.45** 
13. Total Citations per Faculty 1490.00 1479.51 .05 .64** .54** .66** -.29* F i aad .63** ~ baa As .87** 81** -.26 
14. Interdisciplinary Citations per Faculty 1245.13 1247.29 .06 .63** Jott .62** -.24 .76** .64** a" .87** 87**  .78** -.20 1.00** 
15. IO Citations per Faculty 244.87 257.65 -.00 .60** A2** are -.51** -76** att 392% -79** -76** ~——88** -.53** 92%* ggr" 


Note. * indicates p < .05; ** indicates p < .01. Mand SD are used to represent mean and standard deviation, respectively. 


1. Table 2. A list of all programs and all summary statistics. This information is also presented in Tables 
2-8 but sorted and filtered by relevance to particular questions. 

2. Tables 3-4. Rankings of programs by their interdisciplinarity, either raw counts or as a proportion of 
their total output. Higher ranks in Table 3 indicate greater interdisciplinary publication output, 
whereas higher ranks in Table 4 indicate greater interdisciplinary focus relative to all work being 
done there. 

3. Table 5. Rankings of programs by their I-O centrality, that is, the degree to which they are not inter- 
disciplinary, either raw counts or as a proportion of total output. Higher ranks in Table 5 indicate 
greater l-O publication output. 

4. Tables 6-7. Rankings of programs by the raw interdisciplinary citations (Table 6) or proportion of l-O 
interdisciplinary citations (Table 7) they have received. Higher ranks in Table 6 indicate greater 
overall interdisciplinary impact, whereas higher ranks in Table 7 indicate greater interdisciplinary 
impact as a proportion of all impact. 

5. Table 8. Rankings of programs by their I-O influence, that is, the degree to which they are cited by l- 
O publications. Higher ranks indicate greater influence within I-O. 


Importantly, all decisions were based upon Scopus’ internal database, and the reliability of that data- 
base is not perfect. For example, some conference proceedings have been misclassified as journals (and 
vice versa), and it is unclear how the subject area categories Scopus uses were derived. We believe the 
quality to be sufficiently high and consistent that the rankings we created and classifications we devel- 
oped would not be substantially different with perfect reliability; nevertheless, this remains an im- 
portant caveat. Small count and/or ranking differences should be interpreted with this limitation in 
mind. Additionally, it caused the category Catalysis to appear as in Table 10 as the number one non-l-O 
discipline for Virginia Tech because Scott Geller has published 19 articles in Bulletin of the Psychonomic 
Society, a journal which has not published an issue since 1993, and for some reason, Scopus considers 
one of that publication’s categorizations to be Catalysis. From a manual review of disciplinary assign- 
ments, it appears that cases like these are relatively unusual, but this also illustrates why we ultimately 
went with a popularity metric to determine whether a publication was I-O or not rather than using Sco- 
pus’ internal categorization scheme. 


Results 


Although it may seem obvious with the data in front of you, we were surprised at how overwhelmingly 
popular the Journal of Applied Psychology is as an outlet for I-O psychologists in PhD programs, as 
shown in Table 1. Nearly 1 out of every 10 publications by I-O faculty in PhD programs is in JAP, and this 
rate dwarfs all other outlets. By looking at plateaus in Figure 1, additional tiers emerge. Among core I-O 
publications, the second tier of popularity includes Journal of Business and Psychology, Journal of Organ- 
izational Behavior, and Journal of Vocational Behavior. The third tier is larger, including Human Perfor- 
mance through Organizational Behavior and Human Decision Processes. The fourth tier includes Organi- 
zational Research Methods through Educational and Psychological Measurement. Beyond that, an ex- 
tremely large tail appears. Although it is not shown in Figure 1 due to space considerations, l-Os teach- 
ing in PhD programs have published in 1,288 distinct journals over their careers. It is a very long tail in- 
deed, with only one publication, across all I-O psychology faculty, appearing in 568 of those journals. 

For example, only one I-O psychologist ever has published one article each in the journals Humor and 
Zygon.’ Missed opportunities for our field, clearly. 


More broadly, perhaps most striking to us was that there appears to be a general productivity factor 
that crosses disciplinary borders. Faculty/programs who publish more I-O work tend to also publish 
more interdisciplinary work (r = .64). Nevertheless, substantial differences in terms of program focus 
did emerge. For example, the University of Minnesota, which traditionally is ranked near the top of all 
l-O programs in terms of overall productivity, scored at the bottom of Table 4, suggesting those faculty 
much more rarely (32%) publish outside of core l-O psychology journals than within them, although this 
is relative their general elevated level of productivity (i.e., #1 in IO publications per faculty). In fact, 
most traditionally highly ranked programs appear toward the middle or end of that table. Nevertheless, 
many of these programs are highly ranked in terms of raw interdisciplinarity citation impact (Table 6), 
suggesting that highly ranked I-O programs also tend to publish I-O research that fields outside of I-O 
psychology find useful, despite not conducting as much interdisciplinary research themselves, relatively 
speaking. Thus, high-quality research within I-O does appear to be recognized as generalizable to other 
fields, at least to a degree. 


In relation to individual programs, the most interesting interpretive results to us were in examining Tables 
4 and 7 and tracking high interdisciplinarity impact programs in other tables. For example, 91% of the cita- 
tions that Clemson receives come from interdisciplinary publications, with an interdisciplinarity impact ra- 
tio of 13022:1229 (from Table 7), and publications coming from that program are 79% interdisciplinary, 
with a publication ratio of 314:83 (from Table 4). This suggests high impact in literatures outside of |-O 
psychology that may not be evident at first glance in traditional rankings, most specifically in the areas of 
social psychology, psychiatry and mental health, and public health (from Table 10). In terms of overall 
publication rates, Clemson appears lower ranked than would be predicted by their citation rate. This con- 
trast highlights cross-disciplinary differences in impact that are traditionally difficult to interpret. 


Discussion and Takeaways 


There were many interesting takeaways for us within these tables. We do not have the space to de- 
scribe them all, but considering the sheer number of tables and the amount of information we needed 
to leave out of them even so, we thought it might be useful to provide three highlights. First, an emer- 
gent finding for our team that is not evident in the tables is that programs with relatively lower interdis- 
ciplinarity rarely have that interdisciplinary identity across faculty. Instead, individual faculty members 
often represent most or the entirety of an interdisciplinary focus of a program. For example, the second 
most common interdisciplinary publication area for Old Dominion University, where the lead author is 
currently employed, is “Computer Science Applications.” However, the lead author is also the only per- 
son at Old Dominion consistently publishing in this area. Thus, the individual faculty rankings and details 
available in the web app we created (https://tntlab.shinyapps.io/io_programs/) may be more useful to a 
would-be graduate student trying to identify an advisor with a certain interdisciplinary focus than the 
program listing, if that program is not overwhelmingly interdisciplinary. 


Second, we were surprised at how high the interdisciplinary citation counts were (see Table 6) in rela- 
tion to I-O citation counts. All 53 I-O programs listed in the SIOP database have a greater impact on 
fields outside of I-O psychology than on I-O psychology itself in terms of number of publications influ- 
enced. The smallest ratio here was the University of Tulsa, which had 2.56 interdisciplinary citations per 
l-O citation. The greatest difference was found for Alliant University at San Diego, with 33.88 interdisci- 
plinary citation per I-O citation. In general, the small size of I-O psychology means that any one publica- 
tion we produce on average has a much greater impact outside of our field than it does within it. Thus, 


if a prospective graduate student is interested in “impact” in an absolute sense—minds changed, re- 
search influenced—it is even more important to apply to programs with faculty whose work has implica- 
tions beyond I-O. 


Third, we were also surprised to learn how much the productivity and citations rates of “star faculty” 
tend to influence program rankings. For example, the University of South Florida appears to be cited 
within I-O psychology dramatically more than any other program, with a raw count of 6050 citations 
from the core I-O literature, about 2000 greater than either of the next two on the list, Michigan State 
and the University of Minnesota. However, upon investigation of the person-level data (available in the 
shiny app), we discovered that they owe a lot of thanks for that to Paul Spector, who brought in 3077 of 
those citations by himself. This observation led us to investigate the relative impact of “prolific” aca- 
demics within programs more broadly. Although the University of South Florida has several prolific and 
highly cited faculty, this varies widely by institution. By using the shiny app, we can easily identify the 
academics fieldwide most highly cited within I-O psychology articles; in reverse order, the top 10 are 
Paul Spector, Deniz Ones, Paul Sackett, John Meyer, Chockalingam Viswesvaran, Michael Mumford, Rob- 
ert Eisenberger, Eduardo Salas, Ann Marie Ryan, and Steve Motowidlo. If any of these people were to 
leave their programs, the ranking of their program according to I-O citations would immediately and no- 
ticeably change, although by varying amounts. The same pattern is true for any metric of choice, 
whether considering programs citation-wise or publication-wise, within I-O or in general. This suggests 
some volatility in these rankings in relation to individual faculty members; when one prolific or highly 
cited person leaves a program, we would expect that program’s ranking to dip despite what would likely 
be a similar quality of graduate education, and the largest such drops would be in programs with the 
greatest disparity between the least and most prolific/cited faculty members. In addition to highlighting 
the high stakes of hiring or replacing faculty, this reveals the riskiness of using program-level publica- 
tion-based rankings as the sole decision-making tool for undergraduates considering graduate school; 
thus, we again recommend applicants use all rankings as pieces of information that contribute to their 
application decisions, not as focal criteria, and to consider both individual faculty profiles (in the shiny 
app) and program-level metrics.” 


In terms of general, practical recommendations for those applying to graduate school, we recommend 
undergraduates follow a multifaceted decision-making process that weighs the various pieces of evi- 
dence for each program individually. This is a remarkably more complex strategy than the standard “ap- 
ply as highly ranked as your GREs will support” advice that is often provided. Instead, much like needs 
assessment and job analysis, we recommend undergraduates articulate their career goals and how grad- 
uate school figures into them before looking at any rankings. With those goals written down, rankings 
should be chosen that help reach those goals. The mentors that will help a student have maximal im- 
pact within I-O psychology are likely quite different from those that will help them have an interdiscipli- 
nary impact. Considering the general shift of all academic disciplines towards interdisciplinary perspec- 
tive, this will only become more important. 


For those seeking an interdisciplinary perspective in their l-O psychology graduate school experience as 
part of those goals, we recommend a specific process. First, either consult Table 10 for programs that 
contain the interdisciplinary focus of interest or use the online app to find individual faculty with that 
interest. Next, consult the other tables to determine how the programs identified rank on other metrics 
of interest. Finally, consult other non-citation-based rankings, both in this issue of T/P and elsewhere, 
and integrate that information into a holistic picture of a program. In which rankings are your target 
programs strong and in which are they weak, and which balances are attractive to you? Is there more 
than one faculty member with the type of interdisciplinary expertise you’re looking for at any particular 


program? These questions should all be answered before applying—and frankly, a student able to artic- 
ulate a specific rationale for applying to a particular school in a personal statement with such infor- 
mation considered would be impressive indeed. 


Overall, we hope these rankings provide evidence that “impact” is multifaceted to people applying to 
graduate school. There is no clear “best” program or set of programs on all dimensions of productivity, 
and this does not even consider other dimensions of the graduate school experience, such as reactions, 
specific skills gained, employability, cultural fit, and daily workload. Undergraduates should decide what 
is most important to them and consider all programs available to them on those terms. We also hope 
this message extends to those evaluating grant applications or tenure and promotion cases. When in- 
terdisciplinarity is considered explicitly, the meaning of “impact” changes. Although within-discipline 
and interdisciplinary output and impacts are correlated, they are measurably distinct. With this article, 
its rankings, and our online app, we provide resources for people, particularly those applying to gradu- 
ate school, to determine the degree of a program’s impact on other disciplines that was not previously 
possible. We believe it is only through interdisciplinary efforts that I-O psychology will find itself in a po- 
sition of influence within the broader scientific community, a goal our field has been struggling with for 
decades. The present study serves as the first systematic assessment of interdisciplinarity in l-O, and we 
hope it is only the first such effort as l-O continues to evolve to better meet the needs of both I-O psy- 
chologists and society more broadly. 


Note 


1 Comment from RNL: I really wish | could track how many people Googled “Zygon” as a result of reading this sen- 
tence. Perhaps | could get a pub in Humor out of it. 

2 This paragraph was expanded from the originally published version of this article; we thank a reader for calling 
our attention to some ambiguous language. 
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Previous rankings of I-O graduate programs have often focused on “objective” criteria related to research 
outcomes including publication rate and conference representation. Our ranking is differentiated by rank- 
ing l-O PhD programs according to development opportunities offered to graduate students in three core 
areas: teaching, research, and applied practice. Current PhD students were surveyed regarding the devel- 
opment opportunities available to them in their respective programs. We then weighted these opportuni- 
ties using ratings from current SIOP members with experience in l-O related careers. We found that, under 
this ranking scheme, the most highly ranked schools varied across the three development areas, suggest- 
ing that prospective students should consider their future development and career goals when assessing 
potential graduate programs. Further implications and limitations of our results are discussed. 


An examination of the industrial-organizational (I-O) literature reveals no shortage of rankings of I-O 
PhD programs (e.g., Beiler, Zimmerman, Doerr, & Clark, 2014; Gibby, Reeve, Grauer, Mohr, & Zickar, 
2002; Howard, Maxwell, Berra, & Sternitzke, 1985). Although there are exceptions (e.g., Kraiger & Aba- 
los, 2004), these studies typically rely on metrics such as research productivity, journal prestige, and 
conference representation. These are important aspects of a program and they deserve to be quantified 
and investigated. As pointed out by Salter, Allen, Gabriel, Sowinski, and Naidoo (2016) in their Call for 
Proposals, however, these criteria may be less important for prospective graduate students whose ca- 
reer goals lay outside of research-focused academia. Students interested in applied careers, for exam- 
ple, may desire schools with more opportunities to develop and practice applied skills and thus will 
place less weight on research productivity in their decision making. This is especially pertinent to a field 
like I-O psychology, where approximately half of all SIOP members have a career with an applied focus 
(Silzer & Parson, 2011). Many prospective students may also be interested in an academic career with a 
focus on teaching. Most existing ranking methodologies do not incorporate information relevant to fu- 
ture teachers and therefore may result in students overlooking programs that excel in providing them 
with opportunities more closely aligned with their career goals (Salter et al., 2016). 


Additionally, “objective” criteria such as publications or other indices of research success commonly 
used to rank programs (Beiler et al., 2014) fail to account for current student perceptions. Current stu- 
dents’ judgments are likely more pertinent to the needs of prospective students as the experiences of 
current graduate students are likely to be similar to those of incoming graduate students. To this end, 
Kraiger and Abalos (2004) used current student judgments on 20 different criteria to create rankings 
that are relevant for prospective students. They created an overall ranking, as well as rankings on three 
broad factors (program resources, program culture, and program costs) derived from their 20 criteria. 
The present study utilizes a similar approach by focusing on three broad factors that are important to I- 
O PhD programs. However, we developed ours a priori to focus on factors relevant to prospective stu- 
dents’ career trajectories. Heeding the call of Salter et al. (2016), we use ratings from current students 
to describe I-O PhD programs based on the development opportunities they provide. 


Although current graduate students are an effective source of information on the opportunities and re- 
sources a program provides to graduate students, they are still in training and therefore, likely to have 
limited career experience. In comparison, trained I-O psychologists in academic and applied positions 
are likely to be better judges of the value of various development opportunities for learning the skills 
necessary to perform well in I-O related jobs. We therefore use current SIOP members employed in I-O 
related careers (e.g., consulting, research, academia) as subject matter experts (SMEs) on the im- 
portance of these development opportunities for the three core areas (teaching, research, and applied 
practice) instrumental to many I-O careers. 


Development Opportunities 


We define development opportunities as opportunities that programs provide for students to engage in 
activities and practices that help students gain skills instrumental to careers as l-O psychologists. We cat- 
egorize these opportunities into three broad areas: teaching, research, and applied practice. We 
acknowledge that these areas overlap for many I-O psychologists and that I-O careers often emphasize 
skills in more than one of these areas. This framework, however, allows prospective students to com- 
pare schools across multiple dimensions and make decisions based on the area(s) most related to their 
future career goals. For example, students who wish to pursue a career as a teaching-focused academic 
may weight programs’ teaching and research development opportunity scores more heavily when 
choosing which schools to apply to and which to attend. In contrast, a student who is less certain of 
what I-O career they would like to pursue (e.g., practitioner vs. academic) may choose to apply to 
schools that rank moderately high in all three development areas, rather than giving preference to the 
highest-ranking schools in any one area. 


Teaching development opportunities provide students with opportunities to develop skills in various 
aspects of teaching such as managing classrooms, designing courses, evaluating student work, and relay- 
ing information to students. Programs that rank highly in this area provide students with more varied 
teaching experiences throughout their graduate career such as serving as a teaching assistant or instruc- 
tor of record. We distinguish among several teaching opportunities, including assistantships with admin- 
istrative duties, assistantships that involve teaching a lab or recitation, and serving as an instructor of 
record. A full list of these opportunities can be found in Table 1. Such development opportunities are 
likely to be valued by students pursuing a career in academia, especially those hoping to apply to teach- 
ing-focused schools. 


Research development opportunities are conceptually the most similar to the traditional ranking crite- 
ria described previously. However, we assess opportunities that help students develop the skills neces- 
sary to carry out research rather than measuring the outcomes of research (e.g., journal publications). 
Programs that rank highly in this area encourage research in their students, provide multiple opportuni- 
ties to learn necessary skills and methods to conduct high quality research, and provide collaborative, 
research-focused environments. For example, we examine whether programs provide opportunities for 
funding, collaboration with other researchers, and opportunities for research assistantships. Prospective 
students interested in a career in academia or an applied career with a research focus will likely benefit 
from programs offering more of these opportunities. 


Applied practice development opportunities help students gain experience in applying I-O psychology re- 
search findings to organizations, interacting with organizational stakeholders, and communicating with lay 


Table 1 


Development Opportunities by Category 


Category 
Teaching 


Research 


Applied 


Item 


Opportunities to serve as a teaching assistant with grading 
or administrative duties 

Opportunities to serve as a teaching assistant with duties 
including the instruction of lab or recitation sessions 

Opportunities to serve as an instructor of record 

A graduate level course on teaching or instruction of under- 
graduate students 

Opportunities to gain feedback from faculty on teaching 

Opportunities for guest lecturing 

Opportunities to teach a variety of courses and levels 

Opportunities to network with alumni in the academic 
(teaching) world 


Opportunities to be a research assistant 

Awards for excellence in research 

A graduate level course on up-to-date research methods 
and statistics 

A graduate level course or workshop on grant writing 

Opportunities to apply for internal funding for graduate 
student research 

Opportunities to apply for external funding for graduate 
student research 

Formal mechanisms (e.g., research groups) that encourage 
student involvement in student-driven research 

Formal mechanisms (e.g., research groups) that encourage 
student involvement in faculty-driven research 

Class projects that build research skills (e.g., research pro- 
posals) 

Encouragement of collaboration among researchers in 
other areas, departments, and schools 

Conference presence and encouragement for students to 
attend conferences 

Opportunities to work with faculty and graduate students 
who are productive researchers 

Opportunities to work with faculty and graduate students 
who share your research interests 

Opportunities to network with alumni in the academic (re- 
search) world 


Class projects that build applied skills (e.g., job analysis) 

A graduate level course on consulting and/or applied topics 

Opportunities for consulting through faculty 

An in-house consulting firm 

Opportunities for consulting (other than through faculty 
and in-house consulting firms) 

Opportunities for applied internships or jobs 

Opportunities to network with alumni in the applied world 

Relationships with local businesses 

A graduate level course on methods and statistics used in 
applied work 


Wave 2 SME 
Rating M 


3.33 


3.96 


Wave 2 SME 
Rating SD 


1.17 


95 


1.08 


Note. n = 826. Members of SIOP served as SMEs. 


audiences. Programs that rank highly in this area may provide graduate students with opportunities to work 
on consulting projects overseen by faculty or include courses requiring the completion of projects emphasiz- 
ing skills commonly used by I-O psychologists in applied settings (e.g., job analyses, performance appraisal). 
Prospective students interested in a career as a practitioner or as an academic who also consults inde- 
pendently or oversees student consulting projects may be interested in programs ranking highly in this area. 


We assess the presence of development opportunities offered by programs in each of these three areas. 
In addition, the average earliest year in the program during which these opportunities are offered was 
also factored into the rankings. Programs that offer development opportunities to students earlier in the 
program allow students more flexibility in choosing when to engage in opportunities, as well as more 
time spent engaging in the opportunities relevant to their developmental and career goals. 


The Present Study 


In sum, we use two waves of data collection to rank programs on the extent to which they provide de- 
velopment opportunities in three core areas: teaching, research, and applied practice. In Wave 1, we 
used judgments from subject matter experts (SMEs) in I-O psychology PhD programs to derive a list of 
development opportunities within each area. In Wave 2, we collected information about which develop- 
ment opportunities were offered by their respective programs from students in I-O PhD programs. Judg- 
ments from professionals currently employed in I-O careers were used to weight the importance of 
these development opportunities for gaining skills in each of these three core areas. 


Method 
Wave 1 


First, the first three authors independently generated lists of development opportunities for each of the 
three categories (i.e., teaching, research, and applied practice) before combining and agreeing upon 22 
initial criteria. We then administered these initial criteria to a sample of 32 current I-O PhD students 
from Baruch College, the University of South Florida, and Bowling Green State University. These stu- 
dents served as SMEs for Wave 1. We contacted students from these three programs to ensure a well- 
rounded view of development opportunities to help prevent a single program from dominating the rank- 
ings simply because its students developed the criteria. Participants indicated the perceived helpfulness 
of each criterion for developing skills in the corresponding area on a five-point Likert scale (1 = not at all 
helpful to 5 = extremely helpful). Criteria with mean ratings of three or higher were retained. An open- 
ended item asking for additional criteria and their perceived helpfulness was included for each develop- 
ment area. After examining the item ratings and open-ended responses, a final list of 31 criteria (14 re- 
search, 8 teaching, and 9 applied) was retained for use in Wave 2 (see Table 1). 


Wave 2 


Participants. In Wave 2, we surveyed student affiliates and members of the Society for Industrial-Organiza- 
tional Psychology (SIOP) via the SIOP listserv. SIOP members served as SMEs. A total of 3967 student re- 
sponses and 826 member responses were used. The 32 participants from Wave 1 were contacted as a part of 
this effort. In the student sample, 119 (31%) identified as male, and 203 (53%) identified as female, with an 
average age of 28.20 years (SD = 6.86). The modal year in program was second. In the member sample, 340 
(41%) identified as male, and 302 (37%) identified as female with an average age of 40.86 (SD = 12.26) years. 


Procedures. We created separate surveys for SIOP student affiliates and SIOP members. Students were asked to 
indicate how many development opportunities their current program provided from the list of 31 criteria retained 
from wave one. Each criterion was rated using a 3-point scale (yes, no, and unsure). Only yes responses were con- 
sidered as an indicator that a program provided that opportunity. Students were also asked to indicate which year 
in the program (e.g., first year, second year) each criterion was first offered to students. Last, they were asked to 
indicate to what extent they personally used or engaged in each opportunity offered by their program. This rating 
was made ona 5-point Likert scale (1 = not at all to 5 = a great deal). This last metric was used to examine whether 
students of some programs took advantage of more opportunities than others. These rating did not impact pro- 
gram rankings. To minimize bias, the authors did not provide ratings for their programs. 


The SIOP members survey asked participants to rate how effective each of the retained Wave 1 criteria was 
to developing skills in the corresponding area (e.g., research; from 1 = not at all effective to 5 = extremely ef- 
fective). Although not every SME we surveyed was necessarily an expert in every area, we believe averaging 
across many SME judgments gives a robust assessment of the importance of each opportunity. 


Results 


Table 1 includes means and standard deviations of SIOP member ratings of each criterion. Each program 
received a score for each developmental area (i.e., teaching, research, and applied), resulting in three 
separate indices per program. Scores were calculated through the following steps. First, a weighted sum 
of the number of development opportunities offered in each area was created for each participant. Cri- 
teria were weighted using mean importance ratings from SIOP members. This resulted in three develop- 
ment opportunity scores (teaching, research and applied practice) for each participant. These scores 
were then averaged across students for each program. To help protect against sampling error, only pro- 
gram scores derived from at least four student responses were calculated. This resulted in three devel- 
opment opportunity scores for each program. Last, each score was adjusted by dividing the average year 
in which development opportunities in the corresponding area were offered. Therefore, hypothetically, 
if a program offered every teaching development opportunity in the first year of the program, no adjust- 
ment would be made to the program’s teaching development opportunity score. 


It is worth noting that the correlations between scores adjusted by year and unadjusted scores are high, 
ranging from .83 to .88. However, adjusting by year gives credit to programs that provide opportunities 
earlier. We believe that a development opportunity that is accessible earlier in the program is generally 
more valuable than one that can only be accessed in later years. Students have a limited amount of time 
in graduate school and earlier access to opportunities allows them more flexibility in choosing which op- 
portunities to pursue, more experience with opportunities they are interested in, and more time to re- 
ceive feedback and develop their skills in a given area. The mean, standard deviations, and intercorrela- 
tions for program scores in each of the three areas are provided in Table 2. 


Table 2 
Means, Standard Deviations, and Correlations for Program Scores 
M SD Teaching score Research score Applied score 
Teaching score 13.28 3.60 n=37 
Research score 35.57 10.02 .81* n=40 
Applied score 16.18 5.88 10 31 n=40 


Note. *p < .01. Number of programs with a score in a given category is indicated on the diagonal. 


The full rankings of programs are shown in Tables 3 through 5; we indicate the number of student affili- 
ate responses for each program in parentheses. Given concerns such as sampling error, the difference 
between any two programs close to each other in rankings is unlikely to be practically meaningful. 
Therefore, we recommend that prospective students not pay too close attention to specific ordering of 
similarly ranked programs but instead use the tables to obtain a general picture of the opportunities 
that a program offers relative to the rest of the programs in our sample. 


The adjusted rankings were compared to ratings on depth of involvement in opportunities. Interestingly, 
programs that offered more teaching development opportunities did not have students who were more 
involved in those opportunities, r= .17, p = .30. In contrast, programs that offered more research and 
applied opportunities had students who were more involved in those opportunities, r = .58, p < .001 and 
r = .38, p = .02, respectively. 


Discussion 


The resulting rankings showcase the variety of PhD programs offered in l-O psychology. Although, many 
programs appear within the top 50% for all three development areas, very few programs rank highly 
across all categories. This demonstrates the utility of the criteria used here, as programs’ provision of 
different types of development opportunities appear to be evident to students. Furthermore, it seems 
unlikely that students are uniformly rating their programs highly on every opportunity offered. 


These rankings are useful for prospective graduate students faced with complex decisions about which 
programs to apply to, visit, and attend. Students can use these results to determine which programs 
may be best suited to their ultimate career goals. Students with well-defined career goals (e.g., applied 
practitioner) may choose to place more weight on schools’ rankings in one particular area (e.g., applied 
practice). In contrast, students with less well-defined career goals may opt to apply to programs that 
rank moderately well across all three development areas. It is worth noting that a low ranking on these 
lists is not meant as an absolute judgment of a given program—this is not an exhaustive list of all I-O 
programs and it represents just one method of conceptualizing rankings. 


This study has several limitations. First, it is possible that some students purposefully misrepresented 
their programs (either by artificially inflating or deflating their ratings). As Kraiger and Abalos (2004) 
point out, this is a problem inherent to all subjective ranking criteria. Although such response distortion 
may occur, we agree with Kraiger and Abalos (2004) that, when possible, subjective ratings should be 
used in conjunction with more objective criteria to provide corroborating evidence. Additionally, we re- 
moved participants who indicated that their program provided all possible development opportunities 
and no participants indicated their program had fewer than 4 opportunities. Lastly, the rankings indicate 
that no program is in the top five for all categories, providing evidence that the effects of response dis- 
tortion were likely limited. 


We also adjusted raw scores in each development opportunity based on the earliest year opportunities 
were offered. This rewards programs who provide opportunities to students earlier in their graduate train- 
ing. We acknowledge that there may be some situations where this does not necessarily promote develop- 
ment. For example, offering students the chance to teach a course early in their career without proper 
training may be detrimental to growth as a teacher. Nonetheless, again, we believe that a tendency to of- 
fer development opportunities earlier rather than later is ultimately beneficial to graduate students. As 
mentioned, the correlations between adjusted and unadjusted scores are very high, indicating that this 


Table 3 
Program Rankings in Teaching Development Opportunities 


Ranking Program 
1 Texas A&M University (7) 
2 University of Minnesota (9) 
3 Northern Illinois University (5) 
4 University of Georgia (10) 
5 Michigan State University (11) 
6 George Mason University (16) 
7 Bowling Green State University (9) 
8 Saint Louis University (4) 
9 Auburn University (11) 
10 University of Akron (12) 
11 University of Oklahoma (9) 
12 University of Missouri - St. Louis (12) 
13 University of South Florida (4) 
14 Colorado State University (11) 
15 Old Dominion University (6) 
16 Florida International University (7) 
17 Clemson University (6) 
18 Pennsylvania State University (6) 
19 University of Connecticut (5) 
20 Baruch College (19) 
21 Portland State University (5) 
22 Rice University (15) 
23 George Washington University (6) 
24 State University of New York at Albany (6) 
25 University of Nebraska at Omaha (4) 
26 University of Central Florida (6) 
27 Roosevelt University (5) 
28 University of Houston (10) 
29 Louisiana Tech University (16) 
30 Central Michigan University (8) 
31 Florida Institute of Technology (5) 
32 Illinois Institute of Technology (17) 
33 University of Illinois at Urbana-Champaign (6) 
34 The Chicago School of Professional Psychology (4) 
34 Teachers College, Columbia University (6) 
36 Capella University (8) 
37 Grand Canyon University (16) 


Note. Numbers in parentheses indicate sample size of student responses from that institution. Only 
programs with at least four responses are included in this table. 


Table 4 
Program Rankings in Research Development Opportunities 


Ranking Program 
1 University of Minnesota (9) 
2 University of Georgia (10) 
3 Michigan State University (11) 
4 Texas A&M University (7) 
5 Pennsylvania State University (6) 
6 Old Dominion University (6) 
7 Florida International University (7) 
8 University of South Florida (4) 
9 University of Houston (10) 
10 Colorado State University (11) 
11 Roosevelt University (5) 
12 Northern Illinois University (5) 
13 Portland State University (5) 
14 University of Nebraska at Omaha (4) 
15 University of Missouri - St. Louis (12) 
16 Auburn University (11) 
17 Saint Louis University (4) 
18 George Mason University (16) 
19 Bowling Green State University (9) 
20 Louisiana Tech University (16) 
21 University of Connecticut (5) 
22 Clemson University (6) 
23 University of Oklahoma (9) 
24 Baruch College (19) 
25 Rice University (16) 
26 Central Michigan University (8) 
27 University of Akron (12) 
28 George Washington University (6) 
29 Teachers College, Columbia University (8) 
30 State University of New York at Albany (6) 
31 Illinois Institute of Technology (17) 
32 University of Central Florida (7) 
33 University of Illinois at Urbana-Champaign (6) 
34 Florida Institute of Technology (5) 
35 The Chicago School of Professional Psychology (4) 
36 University of Phoenix (6) 
37 Walden University (7) 
38 Alliant International University (5) 
39 Grand Canyon University (17) 
40 Capella University (9) 


Note. Numbers in parentheses indicate sample size of student responses from that institution. Only programs 
with at least four responses are included in this table. 


Table 5 
Program Rankings in Applied Development Opportunities 


Ranking Program 
1 Louisiana Tech University (16) 
2 Pennsylvania State University (6) 
3 Roosevelt University (5) 
4 The Chicago School of Professional Psychology (4) 
5 Illinois Institute of Technology (17) 
6 Alliant International University (4) 
7 Michigan State University (11) 
8 University of Nebraska at Omaha (4) 
9 University of Houston (10) 
10 Bowling Green State University (9) 
11 Clemson University (6) 
12 Saint Louis University (4) 
13 University of South Florida (4) 
14 Old Dominion University (6) 
15 University of Akron (11) 
16 Portland State University (4) 
17 Florida Institute of Technology (5) 
18 University of Georgia (10) 
19 George Mason University (16) 
20 Rice University (14) 
21 University of Minnesota (9) 
22 Colorado State University (11) 
23 Northern Illinois University (5) 
24 University of Connecticut (5) 
25 University of Central Florida (6) 
26 State University of New York at Albany (6) 
27 George Washington University (6) 
28 Texas A&M University (7) 
29 Florida International University (7) 
30 University of Oklahoma (9) 
31 University of Missouri - St. Louis (12) 
32 Central Michigan University (8) 
33 Teachers College, Columbia University (6) 
34 Grand Canyon University (16) 
35 University of Phoenix (6) 
36 Auburn University (11) 
37 Walden University (6) 
38 Capella University (8) 
39 Baruch College (19) 
40 University of Illinois at Urbana-Champaign (6) 


Note. Numbers in parentheses indicate sample size of student responses from that institution. Only pro- 
grams with at least four responses are included in this table. 


adjustment does not greatly affect rankings. However, it does result in some changes to rank ordering. In- 
terested scholars can contact the corresponding author to obtain the unadjusted rankings. 


The median number of students admitted to an I-O PhD program each year is about four (Tett, Walser, Brown, 
Simonet, & Tonidandel, 2013). Given the relatively small number of PhD students who are likely to be in a pro- 
gram at any given time, our rankings are subject to sampling error. Fortunately, the programs in the top 50% of 
each category had an average of 8.37 responses each. Nonetheless, rankings for programs with small sample 
sizes should be viewed with some caution. In addition, some subjectivity is expected in ratings. Averaging across 
student ratings may hide substantial variability in how students perceive, experience, and utilize development 
opportunities in a program. Different students will likely have somewhat qualitatively different experiences in 
the same program. Despite this, we found that, on average, raters tended to demonstrate high within-program 
agreement on scores in each development opportunity area (average ICC(2) = .93). 


Conclusion 


In this study, we focused on parsimonious and broad criteria with prospective I-O PhD students as the 
intended audience. In general, it is important to note that no single ranking system is perfect. There are 
numerous criteria on which programs can be ranked and these criteria may differ in importance depend- 
ing on who is using the rankings (e.g., current faculty, prospective students). The source of data may also 
affect rankings—perceptions of current students may differ from those of current faculty. It would be 
advantageous for prospective students to use our rankings in conjunction with other criteria (e.g., match 
with faculty research interest, publication rate, location). Many previously published rankings (e.g., 
Beiler et al., 2014; Gibby et al., 2002; Kraiger & Abalos, 2004) and other rankings created in response to 
the Call for Proposals by Salter et al. (2016) provide useful information on I-O programs and should also 
be consulted by prospective students. Overall, ranking |-O PhD programs on development opportunities 
provides a practically useful complement to other ranking methodologies. 


Note 


t We removed nine participants who indicated that their program provided all possible development opportunities, as 
this could indicate artificial rating inflation. These nine participants were each enrolled in a different I-O program. 
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Graduate program rankings occur within the industrial-organizational psychology community on a fre- 
quent basis. These rankings are calculated in a myriad of ways: research productivity of faculty (Levine, 
1990; Winter, Healey, & Svyantek, 1995), number of faculty serving on editorial boards (Jones & 
Klimoski, 1991), number of student conference presentations (Payne, Succa, Maxey, & Bolton, 2001; 
Surette, 1989, 2002), expert opinions such as U.S. News and World Reports (1995; 2001), and finally - 
student perceptions (Kraiger & Abalos, 2004). Although all of the aforementioned methods provide valu- 
able information to both incoming students and current faculty alike, we suggest that student percep- 
tions provide a unique insight that can add value to the ranking of graduate programs, particularly for 
prospective I-O graduate students. 


Kraiger and Abalos (2004) suggested that the perspective of current graduate students is both important 
and necessary. They used graduate student opinions to create a set of criteria, ratings of programs 
based on that criteria and in turn, rankings. In doing so, Kraiger and Abalos set several important prece- 
dents in industrial-organizational psychology graduate program ratings. First, they used current gradu- 
ate students as subject matter experts (SMEs) in order to determine which criteria students value the 
most, a novel step in l-O program rankings. Second, they measured the perspectives of students from 
both doctoral programs and master’s programs. This is important because SIOP lists 157 MA and MS 
programs from which potential students can choose, but there is little ranking information available to 
guide these students in their decision besides what is provided on each program’s website or the 
searchable I-O program database on the SIOP website. Having an available and updated source for pro- 
gram comparison based on information provided by students currently at the programs they hope to 
attend would be invaluable to prospective MA/MS students. 


Following their lead, we aimed to conduct student program ratings using similar methodology to Kraiger 
and Abalos (2004) with the goal of updating their previous rankings and offering a unique (student) per- 
spective to the broader effort on program rankings. 


Support and Criticism for Program Rankings 


Program directors, faculty members, and students have a range of stances regarding program rankings, 
ranging from eager to cautious to completely distrustful of the results of any type of program ranking 
method. However, in defense of rankings, it is unwise for any organization to ignore the perspective of 
its customers (“Why customer satisfaction is important,” n.d.). In the case of graduate programs, the 
customers are the students. Prospective students, much like consumers choosing a new product or 
home, are likely looking for product information to make consequential decisions. The ranking of the 
schools can be one such piece of information gathered in an assessment process. It is information that 
prospective students, the consumers, may desire during their decision making processes. 


However, it has been noted extensively that program rankings are not a flawless means by which to 
measure program quality. Kraiger and Abalos (2004) provided examples of ranking flaws: “rankings 
based on program reputation may be unrelated to current faculty productivity given halo (general repu- 
tation of the university), turnover, or raters who do not fully understand the discipline or activities of 


individual institutions” (p.28), demonstrating serious issues that must be considered when interpreting 
rankings. Still, we contend that although rankings may be flawed, they are an inescapable fact of a com- 
petitive academic market. Indeed the very approach we take here, that of rankings based on graduate 
student opinion, are not impervious to error. For example, it is natural for graduate students to believe 
that their program is a top program and rate it accordingly. Confirmation bias tendencies make it more 
likely for students to interpret and recall information that confirms their program is elite because that 
confirms their preexisting beliefs (i.e., that they chose a top program to attend, they made a good deci- 
sion; Plous, 1993). Furthermore, students typically only have experience with one program. Thus, differ- 
ent from a consumer who may have experience with many brands, students only have their own individ- 
ual experience with one individual program, which gives them little basis by which to evaluate it. How- 
ever, despite these considerations, we believe that the information provided by such rankings can offer 
value as they do represent the lived experience of the very population graduate programs serve. 


The Utility of Student Perceptions for Program Rankings 


Using student perceptions in graduate rankings can be particularly valuable for both the prospective stu- 
dents and for the graduate programs who serve them. In terms of the prospective students, as mentioned, 
information gathering about programs prior to making a significant life commitment is natural, yet limited 
comparative information is available that reflects the student experience in programs. Kraiger and Abalos 
(2004) had produced a program ranking by using scores collected from current graduate students on crite- 
ria that were previously deemed important by current graduate students. This method resulted in rankings 
on a diverse grouping of criteria such as student perceptions of faculty support and accessibility, instruc- 
tion quality, balance between applied and academic emphases, and cost of living. These are all factors pro- 
spective students may consider when choosing a program for the next stage of their academic career. Stu- 
dents have to make decisions that can influence the rest of their life and therefore information about pro- 
grams (both positive and negative) may aid them during this process. 


However, ranking based on student perceptions are not just potentially valuable to the prospective stu- 
dents shopping for programs. Student perceptions are also important for the graduate programs them- 
selves. Because these rankings are based on criteria current students deem as important, these rankings 
can be used by programs as a way to develop areas of their program to better target prospective students. 
For example, programs can leverage the findings from the ratings to build a case to administration for pro- 
gram development purposes. For example, programs may ask for more internally funded research or 
teaching positions for graduate students if these areas are rated particularly low relative to other pro- 
grams (Kraiger & Abalos, 2004). In addition, if programs are able to use the rankings to secure better fund- 
ing opportunities for students, it should be beneficial for recruiting future applicants to the program. 


Method 


The authors of this article followed a similar methodology to the one used in Kraiger and Abalos (2004) 
to (a) determine the criteria graduate students deem important, (b) weigh the importance for those cri- 
teria, and (c) compute the total score for each program. This information was then used to determine 
the overall ranking of programs. 


This project was conducted in two phases: a criterion development phase and an importance and rating 
phase. Both phases of the data collection were in conjunction with another project on student percep- 

tions from Bowling Green State University. The initial criterion development phase was conducted with 
current I-O graduate students (N = 46) listing the criteria they used to evaluate or choose a graduate 


program (e.g., research interests of faculty, location, availability of funding). This phase had student re- 
sponses from 3 different universities and colleges, with approximately two-thirds of students enrolled in 
PhD programs and the remaining one third enrolled in MS/MA programs. Students responded to the 
open-ended question, “List any and all criteria used when selecting your graduate program or that you 
use when recommending a program to another person.” The resulting responses were combined into 25 
criteria and definitions of the criteria were written. See Table 1 for the final list of 25 variables. 


Table 1 
Graduate Student Criteria 


Criteria Definitions 


Application process 
Alumni network 


Admission requirements 
Success of alumni and connection of alumni to the program 


Class offerings 
Cost 


Facilities available 


Faculty quality/expertise 


Topics of interest offered and class times offered 

Tuition, fees, and program-related expenses incurred due to pro- 
gram attendance 

Labs, office space, technology options, statistical packages, jour- 
nals access 

Quality of class instruction, salience of research advice, depth and 
breadth of faculty knowledge 


Faculty productivity 


Quality and quantity of graduate faculty journal publications and 
conference presentations 


Faculty research interests 


Major professor and other professors with research interests simi- 
lar to yours 


Teaching opportunities 


Availability for students to student teach, lecture classes, serve as 
a teaching assistant 


Funding resources 


Financial package available to student, relationship between sti- 
pend amount and cost-of-living 


Graduation requirements 


Requirements are reasonable and match with student's goals (e.g., 
having internship requirements when a student is interested in go- 
ing applied) 


Internship opportunities 


Availability of suitable internships to the program's students 


Job/Internship placements 


Successful placements of current students and alumni in appropri- 
ate internships and jobs 


Location 


Geographic qualities around the campus, access to nearby job/in- 
ternship opportunities, cost of living 


Opportunities for applied projects 


Availability do consulting projects and other types of applied work 
as part of the program 


Program culture 
Program flexibility 


Program ranking 


Program reputation 


Atmosphere of the program, norms, collaborative vs. competitive 
Opportunity for student to arrange their schedule to fit other fac- 
ets of their life, take a semester off for life events 

Knowledge of a program's current ranking (e.g., U.S. News & World 
Report) 


Knowledge of a program's reputation in I-O 


Quality of life/fit and social rela- Social fit with and relationships between students within the pro- 
tionships between grad students gram 


Research opportunities Availability for students to engage in research that relates to their 
topics of interest within the program 

Student productivity Quality and quantity of graduate student journal publications and 
conference presentations 

Student support by faculty/ Mentoring availability and formal and social interactions between 

department the student and the faculty/department. Faculty accessibility to 
students. 

Teaching model used Balance between applied and academic focus 

Learn practical skills Relevant skills are learned by students that will be useful in I-O in- 


ternships and jobs. 


The second phase, the main data collection which involved collecting importance scores and ratings, in- 
volved a widely distributed survey. Data collection for this phase began in September of 2017. The sur- 
vey was administered to current graduate student participants from I-O MA/MS and PhD programs 
across the country. The researchers distributed a Qualtrics survey administered to all affiliates of SIOP 
through the SIOP listserv. To augment the number of student participants for this study, particularly 
MA/MS students, SIOP distributed the study link to |-O program directors so that they could email it di- 
rectly to their students. 


Respondents were told that the purpose of this study was “to collect perceptions of the quality of the 
graduate programs from the perspective of their customers—the graduate students.” Participants were 
asked to rate the importance of the 25 criteria, in general, from their perspective as a current graduate 
student. A sample question for class offerings asked participants to rank the importance of class offerings 
for choosing a graduate school from your perspective as a current graduate student: “topics of interest of- 
fered and/or the class times that courses are offered.” Ratings were collected on a four-point Likert-type 
scale (1=not important at all to 4 = very important). The highest possible score for programs was 100. 


Students were then also asked to rank the quality of their program on each of the 25 criteria. A sample 
item asked “Please provide your perceptions on how the following variables relate to the quality of the 
graduate program in which you are currently enrolled. Only answer this section if you are currently en- 
rolled as a student in a graduate program (e.g., Class Offerings).” Ratings for that variable were col- 
lected on a four-point Likert-type scale (1 = extremely poor class offerings to 4 = extremely good class 
offerings). With the exception of two items (i.e., application process, graduation requirements) that had 
3- point Likert scales, all other items were ranked by participants using a 4-point Likert-type scale with 
anchors tailored to the specific item. Students also provided demographic information. All surveys were 
completed anonymously. 


Notably, students from both MA/MS and PhD programs were included and provided importance ratings 
and rankings on the criteria. We expected that some criteria items may be differentially important to 
master’s students versus doctoral students but allowed the importance ratings to elucidate that infor- 
mation for us. Study respondents from each type of program dictated through the importance ratings, 
which items were key drivers to them. See Tables 2 and 3 for the list of which variables were included in 
the importance ratings for PhD and MA/MS programs respectively. 


Table 2 
Criteria for Calculating PhD Rankings 


Criteria included PhD rankings 


Importance 
rating 


Criteria not included for PhD rankings 


Alumni network 


-0.050109499 


Application process 


Class offerings 


0.283953825 


Facilities available 


Cost 


0.395308266 


Graduation requirements 


Faculty quality/expertise 


1.917152296 


Internship opportunities 


Faculty productivity 


0.061244943 


Location 


Faculty research interests 


1.10055306 


Program flexibility 


Funding resources 


1.026316766 


Program ranking 


Job/Internship placements 


0.692253443 


Student productivity 


Learn practical skills 


1.211907502 
0.061244943 


Teaching opportunities 


Opportunities for applied projects 


Program culture 


1.917152296 


Program reputation 


between grad students 


-0.347054675 
0.09836309 


Quality of life/fit and social relationships 


Research opportunities 


0.914962325 


Student support by faculty/department 


1.397498237 


Teaching model used 


0.09836309 


Table 3 
Criteria for Calculating MA/MS Rankings 


Criteria included MA/MS rankings 


Importance rating 


Criteria not included for 
MA/MS rankings 


Application process 


-0.753117766 


Faculty productivity 


Alumni network 


0.329699052 


Program ranking 


Class offerings 


1.001792249 


Student productivity 


Cost 


0.591068628 


Teaching opportunities 


Facilities available 


-0.902471809 


Faculty quality/expertise 


1.860578 


Faculty research interests 


-0.454409678 


Funding resources 


0.030990964 


Job/internship placements 


1.412515869 


Learn practical skills 


2.009932044 


Location 


-0.043686058 
0.703084161 


Opportunities for applied projects 


Program culture 


1.07646927 


Program flexibility 


-0.93981032 
0.217683519 


Program reputation 


tween grad students 


Quality of life/fit and social relationships be- 


-0.155701591 


Research opportunities 


-0.977148831 
0.964453738 


Student support by faculty/department 


Teaching model used 


0.66574565 


Importance ratings followed by student rankings on the program criteria were then used to determine 
the rankings of programs based on what graduate students deemed important (giving criteria with 
higher importance more weight) and how they rated their program. To calculate the overall program 
rankings, we separated PhD and masters programs and developed separate weights for each program 
type. We included all criteria that were above the mean level of importance rating as well as all criteria 
that were one standard deviation below the mean level of importance rating. By doing this we were able 
to exclude criteria that the sample did not rate as important (9 criteria for PhD students, 4 criteria for 
MA/MS students were excluded from ranking calculations). See Tables 2 and 3 for the list of which varia- 
bles were included in the importance ratings for PhD and MA/MS programs respectively. After obtaining 
the mean importance ratings for each item, we calculated a weight for each criterion. The weight was 
the importance rating for each criterion subtracted from the mean importance rating of the total criteria 
set, divided by the standard deviation of the importance ratings. The weights for each criterion were 
then multiplied by the mean rating on that criterion from each program. The sum of all of the weighted 
criteria provides the rank for each program. 


This study received institutional review board approval and approval from the Institutional Research 
Committee at SIOP. SIOP was instrumental in facilitating the collection of data for this study. The Institu- 
tional Research Committee required that to protect the identity of participants, programs with fewer 
than four respondents could not be analyzed for importance ratings or rankings. Therefore, programs 
with students who did not wish to participate or program directors who did not distribute the study in- 
formation to their students were not included in these analyses. We can make no assertions as to the 
quality or hypothetical ranking of such programs. Simply because a program is not included on our list 
does not imply that it is not a quality program. 


Results 


Ratings were obtained from 1,049 current PhD and MA/MS students. We received data that met our in- 
clusion criteria of having Ns greater than or equal to 4 from 44 PhD programs and 48 MA/MS programs. 
The SIOP website states that there are 78 psychology PhD programs and 125 PhD programs total (which 
includes those programs housed in psychology departments, business departments, etc.). This website 
also states that there are 157 MA/MS programs. There were some programs from which we did not re- 
ceive data, which could be due to various reasons (e.g., students are not current SIOP members and did 
not receive the notification about the study, program directors chose not to forward the study recruit- 
ment information on to their students). There were a few programs in which we did not receive enough 
data to include their program in the rankings, which was unfortunate. We chose to report the top 20 
PhD and MA/MS programs respectively for two reasons: (a) to showcase those truly exceptional pro- 
grams and (b) to shield programs that had strong participation in this study but were scored less favora- 
bly than other programs by their students. 


MA/MS (N = 583) student respondents were primarily female (N = 340), with N = 190 being male and N 
= 9 selecting either prefer not to answer or not providing a response on this item. The average age of 
these respondents was 26.26 (SD = 6.02). Of the MA/MS students surveyed 66% were Caucasian, 9% 
were Asian, 8% selected Other, 8% were Hispanic/Latino(a), 5% were Black or African American, and 2 
respondents were American Indian or Alaskan Native. On average the MA/MS students in our study had 
an average start year of 2016 (SD = .70), had been in their current program for approximately one and 
half years (M = 1.53, SD = .46) years, and estimate their degree completion in 2018 (SD = .59). 


Similar to the MA/MS sample, of the responding 466 doctoral students, N = 276 were female, N = 161 
were male, and N = 18 selected either prefer not to answer or did not provide a response on this item. 
The average age of these respondents was approximately 29 years old (M = 28.85, SD = 7.41). The racial 
makeup of the PhD students was also analogous to the MA/MS students, and 68% of PhD students sur- 
veyed were Caucasian, 9% were Asian, 5% selected Other, 5% were Hispanic/Latino(a), 4% were Black or 
African American, and one respondent was American Indian or Alaskan Native. The PhD students reported 
an average start year of 2014 (M = 2014, SD = 1.98), had been in their current program for more than 3 
years (M = 3.32, SD = 1.75) years, and estimate degree completion in 2019 (M = 2019, SD = 1.42). 


Table 4 

Top 20 PhD Overall Based on Student Rankings 
Rank Program N Raw score Z-score 
1. Portland State University 5 38.348 1.721 
2. Pennsylvania State University 10 38.064 1.615 
3. Michigan State University 10 37.629 1.453 
4. Texas A&M University 8 37.582 1.435 
5. Old Dominion University 9 36.766 1.169 
6. University of South Florida 4 36.759 1.128 
qT: Rice University 14 36.263 .943 
8. University of Georgia 12 36.052 .864 
9. Columbia Teacher’s College 9 35.830 .781 
10. George Mason University 22 35.772 .760 
11. Louisiana Tech University 18 35.484 .652 
12. Wayne State University 6 35.452 .640 
13. University of Minnesota 7 35.406 .623 
14. Northern Illinois University 5 35.018 .478 
15. University of Houston 10 35.012 .476 
16. Seattle Pacific University 10 35.002 .472 
17. University of Missouri-St. Louis 11 34.989 .467 
18. University of Oklahoma 9 34.820 .404 
19. Florida International University 7 34.819 .404 
20. University of Akron 15 34.739 374 

Table 5 

Top 20 MA/MS Programs Overall Based on Student Rankings 
Rank | Program N Raw score Z-score 
1 Xavier University 19 32.107 1.865 
2. University of Tennessee at Chattanooga 18 31.764 1.706 
3. Appalachian State University 13 31.058 1.379 
4 New York University 4 30.619 1.176 
5. Middle Tennessee State University 25 30.597 1.166 
6-tie. | San Diego State University 5 30.329 1.041 
6-tie. | University of Maryland, College Park 12 30.329 1.041 
8. George Mason University 15 30.027 .902 
9. Missouri State University 21 29.874 .831 
10. Columbia Teacher’s College 21 29.785 .789 


11. University of Akron 8 29.778 .786 
12. Radford University 19 29.737 .767 
13. Hofstra University 11 29.614 .710 
14. Florida Institute of Technology 6 29.526 .670 
15. Minnesota State University—Mankato 17 29.489 .652 
16. Chicago School of Professional Psychology 17 29.448 .633 
17. University of Guelph 8 29.058 .453 
18. University of Georgia 18 28.947 401 
19. Wayne State University 5 28.756 .313 
20. Indiana University-Purdue University Indianapolis | 6 28.717 .295 


In an effort to highlight the specific criteria that are most important to I-O graduate students and to pro- 
vide more information on certain program characteristics that may be of interests to faculty and stu- 
dents we provide that information here. The top five criteria PhD students ranked highest were faculty 
quality/expertise (M = 3.83), program culture (M = 3.70), student support by faculty/department (M = 
3.69), learn practical skills (M = 3.64), and faculty research interests (M = 3.61). The top five criteria 
MA/MS students ranked highest were the opportunity to learn practical skills (M = 3.85), followed by 
faculty quality/expertise (M = 3.81), job/internship placements (M = 3.69), program culture (M = 3.60), 
and class offerings (M = 3.58). The following tables below are the program rankings based on the top 
three criteria that were rated as highly important to both PhD and MA/MS students. 


Table 6 
Rankings of Program Culture 

Rank PhD programs Score | Rank MS/MA programs Score 

1. Old Dominion University 3.89 1-tie. Indiana University-Purdue Uni- | 4.00 
versity Indianapolis 

2. University of Akron 3.87 1-tie. Keiser University 4.00 

3-tie. Wayne State University 3.83 1-tie. University of Guelph 4.00 

3-tie. Clemson University 3.83 4. Xavier University 3.95 

5-tie. Pennsylvania State University 3.80 5. George Mason University 3.88 

5-tie. Portland State University 3.80 6. Appalachian State University 3.85 

7. University of Oklahoma 3.78 7. Florida Institute of Technology | 3.83 

8. Texas A&M University 3.75 8-tie. Missouri State University 3.78 

9-tie. Florida International University | 3.71 8-tie University of Tennessee at 3.78 
Chattanooga 

9-tie. Rice University 3.71 10. Carlos Albizu University 3.77 

11. Seattle Pacific University 3.70 11-tie. | California State University, 3.75 
Long Beach 

12. University of Georgia 3.67 11-tie. | Elmhurst College 3.75 

13-tie. | George Mason University 3.64 11-tie. | University of Nebraska at 3.75 
Omaha 

13-tie University of Missouri-St. Louis | 3.64 14-tie. | University of West Florida 3.71 

15. Teachers College, Columbia 3.63 14-tie. | University of Wisconsin-Stout 3.71 

University 

16-tie. | Michigan State University 3.60 16. University of Maryland, Col- 3.69 

lege Park 


16-tie. | Northern Illinois University 3.60 17-tie. | Minnesota State University- 3.67 
Mankato 
16-tie. | Hofstra University 3.60 17-tie. | Missouri University of Science | 3.67 
and Technology 
16-tie. | Chicago School of Professional | 3.60 19-tie. | Hofstra University 3.64 
Psychology 
20. Florida Institute of Technology | 3.58 19-tie. | Middle Tennessee State Uni- 3.64 
versity 
Table 7 
Learn Practical Skills 
Rank PhD programs Score | Rank MS/MA programs Score 
1. Louisiana Tech University 3.94 1-tie. East Carolina University 4.0 
2. Michigan State University 3.89 1-tie. Illinois Institute of Technology | 4.0 
3. Alliant International University | 3.75 1-tie. Keiser University 4.0 
4-tie Pennsylvania State University 3.70 1-tie. San Diego State University 4.0 
4-tie Seattle Pacific University 3.70 1-tie. Seattle Pacific University 4.0 
6. Columbia Teachers College 3.63 1-tie. University of Georgia 4.0 
7-tie Keiser University 3.60 1-tie. University of Nebraska at 4.0 
Omaha 
7-tie Portland State University 3.60 1-tie. University of Texas at Arling- 4.0 
ton 
9. Florida Institute of Technology | 3.55 1-tie. University of West Florida 4.0 
10-tie University of Connecticut 3.50 1-tie. Wayne State University 4.0 
10-tie University of Houston 3.50 1-tie. West Chester University 4.0 
10-tie University of Tulsa 3.50 12. University of Tennessee at 3.94 
Chattanooga 
13. University of Akron 3.47 13. Carlos Albizu University 3.93 
14-tie Illinois Institute of Technology | 3.44 14-tie. | California State University, 3.92 
Long Beach 
14-tie Old Dominion University 3.44 14-tie. | University of Maryland, Col- 3.92 
lege Park 
16-tie Roosevelt University 3.40 16-tie. | Hofstra University 3.91 
16-tie Chicago School of Professional | 3.40 16-tie. | University of Baltimore 3.91 
Psychology 
18. Central Michigan University 3.38 18. University of Maryland, Balti- | 3.90 
more County 
19. Wayne State University 3.33 19-tie. | Minnesota State University - 3.89 
Mankato 
20-tie. | Texas A&M University 3.29 19-tie. | Xavier University 3.89 
20-tie. | University of Guelph 3.29 
Table 8 
Faculty Quality 
Rank PhD programs Score | Rank MS/MA programs Score 
1-tie. Portland State University 4.00 1. Keiser University 4.00 


1-tie. University of South Florida 4.00 2-tie. Indiana University-Purdue Uni- | 3.83 
versity Indianapolis 

3. Michigan State University 3.90 2-tie. University of Tennessee at 3.83 
Chattanooga 

4. Texas A&M University 3.88 4-tie. George Mason University 3.80 

5-tie. Rice University 3.86 4-tie. San Diego State University 3.80 

5-tie University of Minnesota 3.86 4-tie. Wayne State University 3.80 

7. University of Georgia 3.83 7-tie. University of Akron 3.75 

8. Pennsylvania State University 3.80 7-tie. University of Minnesota Duluth | 3.75 

9. Columbia Teacher’s College 3.75 9. Appalachian State University 3.69 

10. University of Missouri-St. Louis | 3.73 10. University of Maryland, College | 3.67 
Park 

11. Florida International University | 3.71 11. University of Baltimore 3.64 

12. Clemson University 3.67 12. Missouri State University 3.62 

13. George Mason University 3.64 13-tie. | Middle Tennessee State Uni- 3.60 
versity 

14. Louisiana Tech University 3.61 13-tie. | New York University 3.60 

15-tie. | Hofstra University 3.60 15. Columbia Teacher’s College 3.56 

15-tie. | Northern Illinois University 3.60 16-tie. | Florida Institute of Technology | 3.50 

15-tie. | University of Illinois at Urbana 3.60 16-tie. | Keiser University 3.50 

Champaign 

18. Bowling Green State University | 3.56 16-tie. | Missouri University of Science 3.50 
and Technology 

19-tie. | University of Central Florida 3.50 16-tie. | Touro College 3.50 

19-tie. | University of Phoenix 3.50 16-tie. | University at Albany, SUNY 3.50 

Discussion 


The overarching goal of this research was to provide valuable information to both incoming students 
and current faculty about I-O graduate programs. We believe that using the perceptions of current stu- 
dents provides unique insight into ranking graduate programs. Regardless of the method for determin- 
ing which program to attend, prospective students are driven to attend the best program possible. As 
we have shown through this research and as evidenced by the other I-O program rankings notated 
above—there are many ways to evaluate said programs. A secondary aim of this research was to pro- 
vide rankings for MA/MS programs in addition to the more frequent PhD program rankings. Even though 
it can be more challenging to find methods for ranking master’s programs, students who wish to attend 
those programs should have a resource similar to students evaluating PhD programs. 


That said, we must acknowledge that in the process of this research we observed that “program rank- 
ing” was not an important criterion for students. It was indeed one of the lowest factors of considera- 
tion for students from both PhD and masters programs. This may indicate that some of our presump- 
tions about the use or practical value of rankings, at least for prospective students, were incorrect. It 
may be that the more fine-grained information such as the ratings of specific criteria is more useful (e.g. 
one student may very much care about funding packages whereas another student may very much care 
about research opportunities). 


Furthermore, we want to also note that the samples collected from each institution may be biased. Stu- 
dents who were either very happy or very disgruntled may be those who were the most motivated to 
respond to our inquiry. Furthermore, students who are more research-focused may have been 
overrepresented here and those interested in teaching or applied work underrepresented as “teaching 
opportunities” was rated as low on importance in these samples as was “internship opportunities” in 
PhD programs. 


We want to underscore that programs that were not included in the top 20 PhD or MA/MS programs 
may also be high quality. Also, any unique method of ranking programs will likely produce different re- 
sults. We realize that this type of research has its drawbacks (e.g. potential biases of the raters as they 
are already in their chosen program) and that not all academics support the use of program rankings. 
We want to underscore that there may be a number of reasons that any given program could have been 
omitted from these rankings but that does not include the program is low in quality. However, we hope 
that this work has provided a useful update to previous attempts at student driven rankings of l-O grad- 
uate programs. 


Ms. Roman and Ms. Barnett wish to thank Dr. Erin Eatough and Dr. Charles Scherbaum for serving as 
faculty advisors for this project as their advice and guidance were crucial. We also wish to acknowledge 
Ms. Stefanie Gisler and Ms. Sabrina Yu for their valuable contributions to the project. 


References 


Americas Best Graduate Schools. (1995, March 20). U.S. News and World Report. 

Americas Best Graduate Schools. (2001, April 9). U.S. News and World Report. 

Jones, R. G., & Klimoski, R. J. (1991). Excellence of academic institutions as reflected by backgrounds of editorial 
board members. The Industrial-Organizational Psychologist, 28(3), 5763. 

Kraiger, K., & Abalos, A. (2004). Rankings of graduate programs in I-O psychology based on student ratings of qual- 
ity. The Industrial-Organizational Psychologist, 42 (1), 28-43. 

Levine, E. L. (1990). Institutional and individual research productivity in I-O psychology during the 1980s. The Indus- 
trial-Organizational Psychologist, 27(3), 2729. 

Payne, S. C., Succa, C. A., Maxey, T. D., & Bolton, K. R. (2001). Institutional representation in the SIOP conference 
program: 19862000. The Industrial-Organizational Psychologist, 39(1), 5360. 

Plous, Scott. (1993). The Psychology of Judgment and Decision Making. p. 233. 

Surette, M. A. (1989). Ranking I|-O graduate programs on the basis of student research presentations. The Indus- 
trial-Organizational Psychologist, 26(3), 4144. 

Surrette, M.A. (2002). Ranking I-O graduate Programs on the basis of student research presentations at IOOB: An 
update. The Industrial-Organizational Psychologist, 40(1), 113116. 

Why customer satisfaction is important (and how to focus on it). (n.d.). Retrieved January 07, 2018, from 
https://www.surveymonkey.com/mp/customer-satisfaction-important-focus/ 

Winter, J. L., Healy, M. C., & Svyantek, D. J. (1995). North Americas top I-O psychology doctoral programs: U.S. 
News and World Report revisited. The Industrial-Organizational Psychologist, 33(1), 5458. 


SIOP 2018-Chicago Conference Highlights 


Tracy Kantrowitz 
Program Chair 


Daisy Chang 
Conference Chair 


The SIOP 2018 conference is almost here! This year’s conference includes many new opportunities for 
learning and networking. Here is your guide to the highlights in Chicago! For additional detail on ses- 
sions and speakers, see the Conference Program article in the January 2018 TIP or use SIOP’s searchable 
program. 


Throughout the Conference and Program: 


Over 980 sessions and posters across the 3- day program 

Over 600 posters across 19 poster sessions 

46 Reproducible Research (RR) presentations that facilitate the sharing of code and techniques 
with attendees; for more information see here: www.siop.org/rr 

59 multidisciplinary sessions that feature collaborations, research initiatives, and unique pro- 
jects with expertise from multiple disciplines beyond I-O 

54 sessions with alternative formats (a new record!) 

14 Communities of Interest (in Mayfair on Thursday and Friday) 

16 Master Tutorials 

7 Debates (ethical assessments, professional practice issues, the future of work, competency 
modeling, employee surveys, feedback on assessments, value of exit interviews) 

SIOP’s mobile app keeps you connected to the conference and gives YOU the ability to provide feed- 
back on sessions (powered by Whova). Please take the time to rate each session you attend! 


New Features to SIOP 2018: 


The “SIOP Select” designation unites special, featured, Theme Track, Alliance, and Executive Board 
sessions under one label to highlight sessions cultivated by SIOP volunteers and committees on 
high demand, high impact topics that are broadly applicable and add value across the member- 
ship. What was previously a series of special, featured, Theme Track, Executive Board, and Alliance 
sessions are now united as SIOP Select! 

“Methods Mania” features the highest-rated methods tutorials and sessions in a one-stop loca- 
tion (Sheraton 4) throughout the program 

The “Reviewer's Choice” block (Friday at 1:00) features the highest rated sessions across a diverse 
array of content areas 


Wednesday 


Preregister now for the excellent set of preconference activities—including informative and inspira- 
tional workshops and consortia. The Newcomer Reception is from 5-6pm in Sheraton 1, and all are wel- 
come to join us in celebrating the start of the conference at the Welcome Reception from 6-8 pm at the 
Ballroom Promenade. 


Thursday 


Opening Plenary: Kick the day off with a must-see opening plenary session. We’ll cheer for award win- 
ners, be introduced to new Fellows, and hear about progress toward the vision of TeamSIOP from Presi- 
dent Fred Oswald. The Dunnette Award address will follow the Opening Plenary, and will feature the 
legendary research and career of Dr. Thomas J. Bouchard. 


The Thursday program includes a fantastic five-session Theme Track, “TeamSIOP Ventures Into New 
Playing Fields,” (chaired by Tracey Rizzuto) in Sheraton 5. The afternoon includes SIOP Select sessions 
on solutions for sexual harassment based on lessons from federal agencies, career strategies from recip- 
ients of practice award winners, and a global perspective on high performance work practices. 


Thursday Evening Networking Reception: Enjoy hors d’oeuvres and network with the top poster winners 
at 6:00pm in Ballroom Promenade. 


Friday 


Join the invigorating Frank Landy 5K Fun Run at 7am on Friday, April 20, 2018 adjacent to DuSable Har- 
bor! 


Friday’s many program highlights include six Friday Seminars on topics including R basics, organizational 
network analysis, innovations in prehire assessment, unobtrusive data sources, creating dynamic data 
visualizations through visual notetaking, and cultivating innovation in organizations. These sessions re- 
quire preregistration and a separate fee from the conference registration, so be sure to sign up first if 
space permits. 


The HR Practitioner Track in Sheraton 3 offers multiple top-rated sessions with applied HR strategies and 
CE credits. Sessions include lessons learned on structuring talent management functions, the impact of 
talent on turnover, performance and profit, HR analytic acumen, driving talent conversations with the 
board, and the influence of technology on human resources management. 


SIOP Select sessions include a “learn by doing” machine learning competition, a virtual debate on the 
future of the field, insights into building a pipeline and sustaining success from winners of science 
awards, advancing women in I-O, linking I-O work to federal policy and funding opportunities, active ag- 
ing at work, and translating the value of I-O. 


Saturday 


Saturday includes fantastic programming that you won’t want to miss, so be sure to stay through the 
close! 


SIOP Select sessions include insights from teaching and early career practice award winners on navi- 
gating a meaningful I-O career, an interactive session highlighting I-O innovations, promoting robust sci- 
ence in I-O, a conversation with SIOP leadership, and a view on assessment practices around the globe. 


4:30: Closing Plenary with keynote address by Mark Squillante, Distinguished Research Staff Member 
and the Area Head of Stochastic Processes, Optimization and Control within the Mathematical Sciences 
Department at the IBM Thomas J. Watson Research Center and Director of the Center for Optimization 


under Uncertainty Research across IBM Research. His address is titled “Decision Making Under Uncer- 
tainty With Applications in Workforce Management.” Mark’s experiences and insights will serve as a fit- 
ting way to close a conference focusing on team and multidisciplinary concepts. 


6:00: Closing Reception: Reflect on a great conference at a Chicago food- and music-themed reception. 
This is a reception you won’t want to miss! 


l-O Practice Events at the Chicago SIOP Conference and 2018 LEC 


Rob Silzer, William Shepherd, Lynn Collins, Mark Morris, and Ben Porr 


Your fellow SIOP members have been working very hard to provide numerous SIOP conference programs, 
courses, and events for I-O practitioners. They have been designing and delivering programs that focus on 
meeting the needs of SIOP practitioners. 


Below are practitioner events that are available to you at the SIOP Conference in Chicago in April and at the 
Leading Edge Consortium in Baltimore in October. We encourage you to take a close look at them and 
participate in them. Please forward this information to other members in your professional network who might 
be interested. 


We need your support and engagement: to continue offering and expanding the services and programs that 
SIOP provides for our practitioners. 


1. Individual Leadership Assessment Course 


There has been a need in SIOP for more advanced professional development courses for practitioners. 
The objective of this course is to provide extended professional training and development of practice 
skills, expertise, and knowledge to those experienced in this specialty. 


If you currently provide Individual Leadership Assessments as a psychologist then we encourage you to 
register for this course. You will learn from the top thinkers in leadership assessment and the course 
will benefit from your interactivity and leadership assessment experience. 


e Module 1: An Overview for Experienced Assessors (Morning session only, Chicago, April 18, 
2018). 

e Module 2: Interviewing Skills for Leadership Assessment (Afternoon session only, Chicago, 
April 18, 2018). 


You can register on the SIOP conference website. 
For course description see http://www.siop.org/Conferences/18con/regbk/workshops/ws11.aspx 
For registration see - http://www.siop.org/conferences/18con 


2. Early Career Practitioner Consortium (ECPC) 


The second annual SIOP Consortium for Early Career Practitioners will be on Wednesday, April 18, 8 
am-5 pm during the SIOP conference in Chicago. The consortium focuses on the career needs and 
aspirations of Practitioners who have up to five years of post-degree (PhD. or terminal Masters) work 
experience. This is a one-day career development experience that includes an overview of employment 
trends and emerging issues related to developing your career, an assessment of your skills along with a 
profile and guidance on developing a career plan, and small group mentoring discussions led by 
experienced I-O psychologist mentors. 


This consortium was a sold out success at last year’s conference. Participation is limited. Interested 
SIOP members can register online! 


See http://www.siop.org/Conferences/18con/Regbk/practitioner consortium.aspx 


3. Speed Benchmarking 


The Professional Practice Committee will be hosting a new event this year at the SIOP conference for 
mid-career and senior level internal / external Practitioners. More experienced I-O Professionals can 
network and benchmark with other Subject Matter Experts facing similar challenges. The benchmark 
topics will include employment surveys. performance management, big data in hiring, legal issues, 
leadership development, hiring contexts, interviewing, and succession planning. See the SIOP website 
for greater details. 


You will have the opportunity to participate in two 30-minute facilitated benchmarking sessions on key 
topics. Speed Benchmarking will take place on Thursday, April 19, 5:00-6:30 PM. Spots are limited, 
please reserve your spot and select two benchmarking topics of your choice 


See SIOP website http://www.siop.org/Conferences/18con/regbk/benchmarking.aspx 
4. Speed Mentoring 


Back by popular demand, the Professional Practice Committee is again hosting Soeed Mentoring. This 
event is designed for early career professionals. There will be limited seats available for student 
participants, as well. This event is a terrific opportunity for Practitioners to seek guidance, knowledge, 
and wisdom from highly seasoned professionals. 


Participants will have the opportunity to participate in two 30-minute roundtable topic-driven 
discussions. Spots for the topic areas are available on a first-come, first-served basis. Soeed Mentoring 
will occur on Friday, April 20, 5:30-7:00. 


If you would like to participate, please sign up on the SIOP website! 
See http://www.siop.org/Conferences/18con/regbk/mentoring.aspx 


5. Practitioner Reception: A Networking Event 


The SIOP Practitioner Reception at last year’s annual conference was a huge success, so we are doing it 
again! The SIOP Professional Practice Committee is hosting the Practitioner Reception—A Networking 
Event, presented by Quintela, on Thursday, April 19 from 6-7:30 pm in the Sheraton II and III 
ballrooms. This event will feature fun activities (e.g., photo booth), Chicago-style food (e.g., deep dish 
pizza, hot dogs, sausages), and free giveaways (first 100 people to complete the networking challenge 
will receive a free drink ticket). 


Don’t miss this great opportunity to network with your practitioner peers! 


6. Great l-O Psychology Practice Debate 


One of the many practitioner-focused sessions at the annual conference will be the Great I-O 
Psychology Practice Debate, which will showcase five leading I-O practitioners who will address critical 
challenges facing I-O practitioners: Are we psychologists? How are PhD level I-O practitioners different 
from master’s graduates and other professionals? The debaters include Rob Silzer, John Scott, Doug 
Reynolds, Jeff McHenry, and Erica Desrosiers. 


Over the last 20 years I-O psychologists have gained great influence and have had significant impact in 
organizations. They are helping organizational leaders be more strategic and make smarter decisions 
regarding human resources. This has resulted in l-O psychologists being identified as one of the top 
growth professions for the future. But will this continue? 


Please join us for a lively and informative debate on core I-O practice issues 


e Friday, April 20, 2018 from 11:30 AM to 12:50 PM, 
e Room Chicago 6, SIOP Conference, Chicago, Grand Sheraton 


See the conference program for other practitioner focused sessions! 


2018 Leading Edge Consortium 
SIOP will offer another major program for I-O Psychology members and Practitioners in 2018. 


The 2018 Leading Edge Consortium will be held October 19-21, 2018 at the Renaissance Baltimore 
Harborplace Hotel 


High Potentials: Identifying, Developing and Retaining Future Leaders 


The Organizing Committee includes cochairs Rob Silzer and Allan Church, and Members John Scott, 
Lorraine Stomski, David Baker, and Raphael Prager. 


This will be a significant event for I-O practitioners who assess, identify, develop, evaluate and retain high 
potential talent. The program will take a talent management approach to HiPos. Look for forthcoming 
program announcements and information on keynote speakers, workshops and other LEC related events. 
This will be a must-attend event for SIOP members working with HiPos and HiPo programs. 


It takes a strong commitment to peers for your colleagues to develop and deliver such a high volume of high 
quality programming and events. Take advantage of these superlative programs for your ongoing professional 


development and expand your network at the same time. 


Thank you and see you in the Windy City and the Inner Harbor! 


‘lad 


SIOP Must-See Sessions & Events for Students 


Wednesday April 18" 
Chicago CIOP Consulting Challenge Presentations - http://www.ciop.net/event-2814478 for registration and details 


Newcomer Reception for First-Time Attendees 
Welcome Reception 


Thursday April 19 


Opening Plenary Session 

Dos and Don’t’s: Thriving as PhD, Masters, and Undergraduate Students 

SIOP Select: Translating “I-O Speak” in Multidisciplinary Teams 

|-O for the Greater Good: Funding your Science with the National Cancer Institute 
Careers Poster Session 

The Road Not Taken: Career Choices in I-O 

Empowering the I-O Internship: How to Maximize Professional Experiences Early 
Networking Reception & Top Posters 


Friday April 20% 


The Utility and Trustworthiness of I-O Literature: Believe it or Not! 

SIOP Select: Building a Pipeline and Sustaining Success as an I-O Scientist 

Due for an Upgrade? The Future of I-O Psychology and HR in the Digital Era 
Bridging the Academic-Practitioner Divide to Improve I-O Science 

Do We Stand? Alternative Methods of Ranking I-O Graduate Programs 

The Great I-O Psychology Practice Debates: Addressing Critical Professional Issues 
Facilitators and Barriers to Academic Careers for Women in STEM 

SIOP Select: Advancing Women in I-O: From Obstacles to Tangible Solutions 

Lost in Progression: Transitioning from the Classroom to Practice 

School’s Out Forever: The Transition from Student to Professional 

What | Wish | Knew when | Picked my Career Path: Advice from Midcareer Professionals 
SIOP Select: I-O Value (No Longer) Lost in Translation 

Speed Mentoring (Preregistration is required) 


Saturday April 21th 


From Student to Intern: Advice and Networking with Professionals 

SIOP Select: Navigating a Meaningful I-O Career: Insights from Award Winners 
Lost in Translation: Getting the l-O Science Message Across to HR Practitioners 
Tales of Betrayal: Insights from I-O Professionals with Nontraditional Career Paths 
Early Career Lessons: Things | Wish | Knew (A Tribute to Chicago Based Movies) 
Closing Plenary Session 

Closing Reception 


SIOP 2018 


CHICAGO APRIL 19-21 


Sheraton | 


Ballroom Promenade 


Chicago VI-X 
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Sheraton 5 
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Riverwalk 
Superior A 
Michigan A 
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Chicago 10 
Chicago 8 
Erie 

Gold Coast 
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Wrigleyville 
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Chicago 10 
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Chicago VI-X 
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5:00 PM 
6:00 PM 


8:30 AM 
10:30 AM 
10:30 AM 
1:30 PM 
3:30 PM 
4:00 PM 
5:00 PM 
6:00 PM 


8:00 AM 
10:00 AM 
10:00 AM 
10:00 AM 
11:30 AM 
11:30 AM 
1:00 PM 
3:00 PM 
3:00 PM 
4:00 PM 
4:00 PM 
5:00 PM 
5:30 PM 


8:00 AM 
10:00 AM 
11:30 AM 
11:30 AM 
1:30 PM 
4:30 PM 
6:00 PM 


6:00 PM 
8:00 PM 


10:00 AM 
11:50 AM 
11:50 AM 
2:50 PM 
4:20 PM 
4:50 PM 
5:50 PM 
8:00 PM 


9:20 AM 
10:50 AM 
11:20 AM 
11:20 AM 
12:50 PM 
12:50 PM 
1:50 PM 
3:50 PM 
3:50 PM 
4:50 PM 
4:50 PM 
5:50 PM 
7:00 PM 


9:20 AM 
10:50 AM 
12:20 PM 
12:40 PM 
2:20 PM 
5:30 PM 
8:00 PM 


Starts Monday at noon, Presentations Wednesday at 5:00 PM 
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Alternative Session Type 
Special Event 
Alternative Session Type 
Poster 

Alternative Session Type 
Alternative Session Type 
Special Event 


Panel Discussion 

Special Event 

Panel Discussion 

Panel Discussion 
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Debate 

Poster 

Special Event 
Roundtable Discussion 
Panel Discussion 
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Alternative Session Type 
Special Event 
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Panel Discussion 
Alternative Session Type 
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Special Event 


Schedule subject to change. Last updated 3/8/2018. This resource was prepared by volunteers from the SIOP Education and Training Sub-Committee for Graduate Student Resources. It does not reflect endorsement by SIOP. 


Full registrants of the conference may attend any of the events below for no additional charge. Guests may attend general conference receptions with an additional fee. More info at www.siop.ora/conference. 


Advice for Student First-Time Attendees 
Before 
Set a goal. Are you there to learn, connect, or find a job? Let your answer guide your decisions at SIOP. 


Make a priority list of three things (sessions, symposiums, panels, etc) a day that you would like to attend. If you 
have energy and interest, then attend more. Tips for choosing sessions: 


e Look for sessions that are relevant to a current project or paper. 

e Ifyou don’t have a set research area, attend sessions that interest you as possible specialties. 

e Check out SIOP’s website for the one-pager highlighting sessions of interest for graduate students. 

e Consider the room location and the time you have between sessions if you choose any back-to-back 
sessions. 


Prepare to connect. Bring business cards and update your social media to put your best foot forward. Download 
the Whova app and sync it to your LinkedIn profile. You can use the app to talk to people who went to the same 
sessions as you. 


Pack accordingly. If you’re not staying onsite, consider bringing some items to help you make the most out of 
the conference. 


e Anotepad and pens so you can jot down names of presenters to look up or connect with later. Also use 
these to prepare questions before or during talks so you can actively participate in the conversation 

e Consider a business card holder, depending on your goals for the conference. You’ll want to make sure you 
don’t lose cards you receive from others. 

e Alight sweater or other professional outerwear, sometimes conference rooms can be chilly! 

e Sandals or other comfortable shoes that fit in your bag, for walking longer distances, if you'll be wearing heels. 


Plan to make the most of your time there. Are there places you’ve been wanting to visit in Chicago? Make a list 
of local restaurants or tourist spots you can visit and set aside at least a little time. 


During 
Check out the poster sessions: It’l] allow you to speak directly with the researcher(s) and share ideas. 


Dress appropriately in professional business attire but not to the point where you'll be uncomfortable all day. 
Pro tip: Always make sure to break in your shoes ahead of time! 


e By the way, we’ve noticed the first day has more formal wear and clothing becomes slightly more casual 
throughout the conference while remaining business-style. 


Decompress. Take time for yourself; the conference can take a lot of energy out of you. It’s ok to take breaks! 


Don’t forget to eat! Be sure to make it to coffee breaks on time to snag snacks, or you can bring some granola 
bars in your bag if you’re worried about rushing between sessions. You can also bring a reusable water bottle to 
take advantage of the water coolers around the conference. Also check out the SIOP receptions for food (and 
networking). 


Remain professional in your attitude and composure, whether it’s in a session, an onsite restroom, or the hotel 
bar. Carry yourself in a professional manner at all times—the I-O field is a small one! 


Network, network, network! 


e Meet up with alumni from your program or current students of a program you’re interested in. These 
connections could help you gain insight or even get a foot in the door. 

e Ifyou go to a session and find someone’s research or experience especially interesting, try to meet them 
afterwards (if they’re not busy)! Chances are, they'll appreciate your interest. 

e Sport the “first time attendee” ribbon to spark conversations. 

e When you get a business card, write about the questions you have for the person, how you met, and so on 
to help you keep track of who is who. 

e Attend social hours coordinated by SIOP or other sponsors that share the same interests as you. 


Have fun! It’s your first SIOP experience, so soak it all in while you can, without putting too much pressure on 
yourself—you’ll likely be back! 


After: Within 1 week 


Revisit goals. Did you meet the goals you set for yourself prior to the conference? What else might you need to 
do as follow-up in order to reach all of your goals? 


Revisit notes and organize them in a way that helps you reference them for later projects or professional 
development. 


Follow-up with a brief email or LinkedIn connection to everyone with whom you exchanged cards or had a more 
extensive conversation. Thank them for the opportunity to chat, mention anything memorable you learned from 
them, and/or provide anything you promised as a follow-up. 


Organize and submit travel-related receipts, if you get any reimbursements from your department. 


Write a summary of the top three things you learned from SIOP 2018. Share it with your advisor, research 
teams, and whoever else might be interested or benefit from it. Ask them what they learned too! 


Members in the Media 
Barb Ruland 


More people are becoming aware of I-O psychology’s relevance and business benefits. Many SIOP mem- 
bers are helping raise the profession’s visibility, and these are the media mentions we found from No- 
vember 27, 2017 through January 31, 2018. 


We scan the media on a regular basis but don’t catch everything. So, please feel free to send us your 
media mentions. We amplify them on social media and share them in this column. Use the column to 
look for project collaborators, spark ideas for research programs, and keep up with your colleagues. 


The Profession of I-O Psychology 


Mentions of the profession in general educational publications is one sign of I-O psychology’s growing 
visibility. The video featuring Beth Payton from Ohio’s Broadcast Educational Media Commission can be 
a great resource for outreach to schools. Love the bowtie the HR manager wears in the video and the 
fact primary and middle school students are being taught about the field! 


SIOP offers a wealth of resources to help with I-O psychology outreach and education efforts, including 
brochures, an overview video, and the interactive career paths tool. 


Although a Swedish website lists I-O as one of “the 5 Best Jobs You’ve Never Heard Of,” word about the 
profession is getting out. EHS Daily Advisor, a publication for environmental health and safety profes- 
sionals noted that I-O psychologists enjoyed some of the highest salary growth during their most recent 
reporting period. US News & World Report also lists it as one of the “25 Best Jobs That Pay 100k.” 


l-O psychology’s rising star is fueled by broadly useful, specifically applicable insights. As one writer for 
Inc. said, “the business world is turning to social scientists to really help them understand why and how 
people tick. It should come as no surprise that industrial organizational psychology is one of the fastest 
growing professions.” 


A couple examples support the idea that the ability to understand human behavior through data analysis is 
key to I-O’s business value: Community officials in Urbana, IL, drew on I-O psychology methods to help cre- 
ate a successful proposal for a community policing program, and the trustees of the University of New 
Mexico selected SIOP member Garnett Stokes to apply her expertise as president of their institution. 


Gender and Diversity Issues 


The sexual harassment scandals that rocked the country and spawned the #metoo movement brought 
many to seek insights from SIOP’s members. Rebecca Bennett contributed to a discussion of gendered 
interactions in the workplace posted in the Free Library’s Journal of Managerial Issues. 


Lisa Scherer discussed what can cause sexual harassment training to backfire. Amy Cooper Hakim pre- 
dicted that there will be an uptick in workplace training to “ensure that all members of an organization 
treat each other with respect.” 


Caren Goldberg contributed I-O psychology insights to several articles on the subject for The Parallax, 
Bustle, and Elite Daily. Vicki Magley also fielded several requests for expert insight, including NPR’s Mar- 
ketplace, Scientific American, Wired, Business Insider, and the New York Times. 


Jaclyn Jensen advocated for #SponsorHer over #MentorHer for helping to solve sexual harassment in 
the workplace in the editorial for Women’s eNews. Industrial Health and Safety News picked up SIOP’s 
article, “How Organizations Can Really Fight Sexual Harassment.” 


Research conducted by Mikki Hebl and Christine Nittrouer on gender disparities in colloquium invita- 
tions, published in the Proceedings of the National Academy of Sciences, was highlighted in a publica- 
tion for those in higher education. 


l-O Activism 


Two student members, Alise Dabdoub and Samantha Elliott, got early career experience with advocacy 
work and media coverage in the Oklahoma University student paper by organizing a protest against the 
proposed tax increases for universities. 


Rice University’s news service highlighted big data’s power for good in the school’s response to Tropical 
Storm Harvey as the introduction to a piece about the school’s new data science minor, cochaired by 
Fred Oswald. 


Career Advice From and For I-Os 


SIOP members were quoted in business and general publications on a wide variety of career develop- 
ment topics. 


A Wall Street Journal article on making good on a new job that’s a bad fit drew on research by Allison 
Ellis, Talya Bauer, and Berrin Erdogan about how new employee behavior influences organizational sup- 
port they receive. 


Amy DuVernet and Tiffany Poeppelman discussed how and why finding a mentor is good for training 
professionals. 


Kristin Shockley discussed the career drawbacks of remote work and offered several tips for mitigating 
them. 


Ben Dattner’s list of things you should never say to your boss was covered by Business Insider and Long 
Room. He also reminded readers of a roundup article for Money about getting along at work, that liking 
everybody isn’t required for organizational success. 


Allan H. Church cowrote a Harvard Business Review “Managing Up” article advising how to recognize 
and improve a situation with a boss who doesn’t like you. Adam Grant urged managers to express grati- 
tude for the workers’ efforts. 


Members also had suggestions for those in the job market. Amy Cooper Hakim offered job seekers ad- 
vice for connecting with interviewers. Lauren Salomon discussed impression management in a Business 
Insider article that also cited research by Marie Waung and Jeff DiMambro. 


Popular Press Topics 


Adam Grant was cited in a blog post about how to beat procrastination, another on how to model resili- 
ence for children, and one about explaining love to them. Ron Riggio contributed to a widely shared 


roundup article on morality and tribe mentality for USA Today. He also helped explain why Matt Lauer 
looks different to people in the wake of his sexual harassment charges. 


Employee Recruiting and Selection, Personality 


Ken Yusko’s presentation on the use of psychological assessments to provide a winning advantage in 
selection for sports teams was highlighted in a Forbes article. An industry publication for the wood pro- 
cessing, pallet, and sawmill industry quoted Ben Dattner on employment testing. 


Facebook’s People VP uses a term, “disagreeable givers,” Adam Grant coined in Givers and Takers to de- 
scribe one of the qualities the organization seeks in its hiring process. 
Employee Engagement, Motivation, and Turnover 


A Salesforce blog post quoted Grant on how to support resilience, after he gave a presentation at the 
company. Christine Allen argued that the competitive business advantage of resilience justifies the or- 
ganizational and personal investment required to build it, and offered tips for doing so. 


Paul Baard covered communication; different types of motivational drive; how organizational culture trans- 
lates to the bottom line; and what to do when motivation takes a hit in his column, “Motivation Matters.” 


Jennifer Cullen wrote for the Harvard Business Review about flaws in employee engagement survey 
questions that can lead organizations astray. 


Employee Burnout, Work-Life Balance 


Allison Gabriel made a bottom line argument for taking care of human resources while talking about the 
need to prevent burnout in a piece for KJZZ public radio. 


How can rituals help you unhook from work at the end of the day? SIOP member Teresa Amabile’s re- 
search was highlighted in this article about how to leave work at work. 


Leadership, Management, and Organizational Culture 


Allan H. Church coauthored an advice column for leaders on working with underachieving team members, 
highlighting different strategies for different types of performance issues. Ben Dattner urged leaders to 
address failures honestly and in plain language; whereas Adam Grant urged them to express gratitude for 
workers’ efforts. 


Alison Eyring said the real key to successful change management is “enabling and supporting people to 
adapt.” Karl Weick is quoted in a Motley Fool podcast on the need for modern leaders to pay more atten- 
tion to compass points than maps, metaphorically speaking. 


Tomas Chamorro-Premuzic collaborated on an article detailing the organizational dynamics that impede 
high potential programs. 


Don’t forget to include SIOP on your mailing list for notices of media mentions and professional accom- 
plishments. 


Introducing SIOP’s 
Committee for the Advancement of Professional Ethics (CAPE) 


Deirdre Knapp and Joel Lefkowitz 


In April 2017, SIOP’s executive board approved formation of a Committee for the Advancement of 
Professional Ethics (CAPE). In addition to Deirdre Knapp as Chair, the initial committee members— 
representing practice, research and/or academic areas--are Joe Allen, George Banks, James Grand, Paul 
Green, Amy Grubb, and Joel Lefkowitz. Rodney Lowman is involved in an advisory capacity, but his 
primary role will be serving on the recently announced APA Ethics Code Revision Task Force (ECTF). 


For a long time, it probably would have been fair to say that the view of research ethics held by many 
SIOP members was “an unreasonable set of rules or expectations designed by intrusive idealists to make 
our lives more difficult” (Lefkowitz, 2017, p. 1.) and that ethics education, when it was taught at all, was 
taught in ways that communicated, “This is something we unfortunately must require you to do, so let’s 
get it over with as quickly as we can, and then we can move on to the important things” (Macrina, 2014, 
p. xviii). But the zeitgeist has changed. We believe that the board’s action recognizes the increasing 
relevance and greater salience of ethical issues in the field and among SIOP members in their work as 
consultants, researchers, and educators (cf. Payne, Morgan and Allen, 2015; Steiner & Yancey, 2013). 


CAPE’s agenda is to: 


e Develop and/or share resources targeted to I-O graduate education in ethical research and 
practice 

e Develop and/or share resources targeted to I-O continuing professional education in ethical 
matters, including promoting inclusion of ethics-related programming at the annual conference, 
inclusion of ethical decision-making thought pieces (e.g., case studies) in SIOP communications 
(e.g., Newsbriefs, website, TIP), and self-study resources 

e = Consult with the Executive Board on relevant SIOP policies (e.g., conflict of interest) 

e Serve as a source of input and response team to proposed changes to ethics-related standards 
(e.g., the APA ethics code) 


To be clear, the need for this new committee is not borne out of a concern that there are too many I-O 
psychologists behaving badly. Rather, it reflects the reality that we all routinely make professional 
decisions that, explicitly or not, reflect ethical choices. Helping I-O psychologists more explicitly 
recognize and navigate situations requiring such choices can only make us stronger as a profession. 
Raising the profile of discourse related to ethics and promoting mindful consideration of ethical matters 
in professional decision-making as I-O psychologists is the goal of this new committee. We also hope to 
encourage the development of and raise the profile of a new generation of I-O practitioners and 
scholars who are thought leaders in this area. 


At the time of this writing, the committee has met (by teleconference) only once, and we are in the 
process of developing an agenda to implement our objectives. To that end, we would very much like to 
hear from you—with your suggestions for implementation, and/or to volunteer your involvement in the 
committee’s activities. You may do this by reaching out to any member of CAPE. 


We will be publicizing the existence of the committee and soliciting member involvement in multiple 
forums and during the SIOP conference (through ad hoc means; the official program has been finalized). 
So look for announcements from CAPE. See you in Chicago! 
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Foundation Spotlight: Are There Too Many Awards? 
Milton D. Hakel, PhD 
What is the best number of awards for SIOP to present? 
It seems like a simple question. How many are there? Would you believe 21? 


Ok, here’s the list: The Dunnette Prize leads off, because it is the biggest ($50,000) and has just been 
presented for the second time, this year to Thomas Bouchard, and in 2015 to Frank Schmidt. 


There are seven Distinguished and Career Awards: Distinguished Professional Contributions Award, 


Distinguished Scientific Contributions Award, Distinguished Service Contributions Award, Distinguished 


Early Career Contributions—Practice Award, Distinguished Early Career Contributions—Science Award, 
Distinguished Teaching Contributions Award, and the newest, the SIOP Humanitarian Award. 


Then there is a growing collection of eight Achievement and Best Paper Awards: William A. Owens 
Scholarly Achievement Award, M. Scott Myers Award for Applied Research in the Workplace, Raymond 
A. Katzell Award in I-O Psychology, Joyce and Robert Hogan Award for Personality and Work 
Performance, Wiley Award for Excellence in Survey Research, Jeanneret Award for Excellence in the 
Study of Individual or Group Assessment, S. Rains Wallace Dissertation Award, and the Schmidt-Hunter 


Meta-Analysis Award. 


| suppose that the five kinds of Conference Awards should also be counted: Best 
Lesbian/Gay/Bisexual/Transgender (LGBT) Research Award, SIOP Best International Paper Award, Robert 
J. Wherry Award for Best Paper at IOOB Conference, John C. Flanagan Award for Best Student 
Contribution at SIOP, and the SIOP Student Travel Award. 


Ah, but SIOP also awards the Leslie W. Joyce and Paul W. Thayer Graduate Fellowship, as well as five+ 
named scholarships (Lee Hakel Graduate Student Scholarship, Mary L. Tenopyr Graduate Student 
Scholarship, and SIOP Graduate Student Scholarships, George C. Thornton, IIl Graduate Scholarship, 
Irwin L. Goldstein and Benjamin Schneider Graduate Scholarships by the Macey Fund)... 


..Plus seven research grants, namely Douglas W. Bray and Ann Howard Research Grant, Sidney A. Fine 
Grant for Research on Job Analysis, SIOP Small Grant Program, SIOP International Research and 
Collaboration (IRC) Small Grant, Adverse Impact Reduction Research Initiative and Action (AIRRIA) Grant, 


James L. Outtz Grant for Student Research on Diversity, and Hebl Grant for Reducing Gender Inequities 
in the Workplace. New for 2019 will be the Graen Grants for Student Research on Design Thinking and 
New Professionals. 


And the Human Resource Management Impact Awards program, a joint venture of SHRM and SIOP and 
their foundations needs to be counted. 


So, 36 and counting is the best current answer to the “how many are there?” question. 
Too many? Not enough? Stimulated by proposals for additional awards, the questions arose during the 


SIOP Executive Board meeting in January. They are now being considered by the Awards Strategy Task 
Force. Fred Oswald and | have appointed Leaetta Hough (chair), Paul Green (vice chair), Lillian Eby, 


Tara Behrend, Betsy Shoenfelt, Tiffany Poeppelman, and John Scott to advise both the SIOP Executive 
Board and the SIOP Foundation Trustees concerning the awards program. They are considering 
questions such as why have awards at all? What are the purposes for giving them? What is meant by 
“award”? Are grants and scholarships included? Is there an optimal number and distribution of awards? 
How big or small should awards be? What factors make awards desirable? Many subsidiary questions 
are also being considered. 


The Awards Strategy Task Force is now deliberating, and welcomes any comments and questions you 
may provide—write to AwdsStratTF@siop.org. Please note that the SIOP Foundation is a separate 
corporation, made necessary by IRS rules for public charities to distinguish it from other nonprofits, such 
as trade associations (e.g., SIOP). The Task Force will report its recommendations in September. 


Thanks in advance for your input. 


Milt Hakel, President, mhakel@bgsu.edu 

Rich Klimoski, Vice-President, rklimosk@gmu.edu 

Nancy Tippins, Secretary, Nancy.Tippins@tippinsgroup.com 
Leaetta Hough, Treasurer, leaetta@msn.com 

Adrienne Colella, Communications Officer, acolella@tulane.edu 
Mirian Graddick-Weir, Trustee, mirian_graddick-weir@merck.com 
Bill Macey, Trustee, wmacey9@gmail.com 

John C Scott, Trustee, JScott@APTMetrics.com 


SIOP Awards Opening Soon! 


Cindy McCauley, Chair 
Awards Committee 


The call for nominations for SIOP and SIOP Foundation awards, scholarships, and research grants 
opens at the annual conference. Don’t miss this opportunity to submit your research project for an 
award or grant, apply for scholarship funding to help with the expense of carrying out your 
dissertation research, or nominate a deserving colleague for one of SIOP’s Distinguished Awards! 


Below is a list of the available awards, scholarships, research grants, and the Joyce and Thayer 
fellowship. For detailed information on the criteria and documentation required, and to submit your 
nomination or application please click here. 


Distinguished and Career Awards 


e Distinguished Professional Contributions Award 

e Distinguished Scientific Contributions Award 

e Distinguished Service Contributions Award 

e Distinguished Early Career Contributions—Practice Award 
e Distinguished Early Career Contributions—Science Award 
e Distinguished Teaching Contributions Award 

e SIOP Humanitarian Award 


Achievement Awards & Best Paper Awards 


e William A. Owens Scholarly Achievement Award 

e M. Scott Myers Award for Applied Research in the Workplace 

e Raymond A. Katzell Award in I-O Psychology 

e Joyce and Robert Hogan Award for Personality and Work Performance 

e Wiley Award for Excellence in Survey Research 

e Jeanneret Award for Excellence in the Study of Individual or Group Assessment 
e S., Rains Wallace Dissertation Award 

e Schmidt-Hunter Meta-Analysis Award 


Research Grants 


e Douglas W. Bray and Ann Howard Research Grant 

e Sidney Fine Grant for Research on Job Analysis 

e SIOP Small Grant Program 

e SIOP International Research and Collaboration (IRC) Small Grant 

e Adverse Impact Reduction Research Initiative and Action (AIRRIA) Grant 
e James L. Outtz Grant for Student Research on Diversity 

e Hebl Grant for Reducing Gender Inequities in the Workplace 


e Graen Grant for Student Research on the Development of New Professional Knowledge 
Workers — New in 2019 


Scholarships and Fellowship 


e Leslie W. Joyce and Paul W. Thayer Graduate Fellowship 

e Lee Hakel Graduate Student Scholarship, Mary L. Tenopyr Graduate Scholarship, and 
Graduate Student Scholarships 

e George C. Thornton, IIl Graduate Scholarship 

e Irwin L. Goldstein or Benjamin Schneider Graduate Scholarships by the Macey Fund 


The deadline for applications is June 30, 2018. We look forward to reviewing your applications. 
Please email me at mccauley@ccl.org, if you have any questions. 


SIOP UN Committee: Help us Identify How I-O Psychology Informs the Sustainable Development Goals 
Julie Olson-Buchanan, Jenna Van Fossen, Kalaani Young, Kyle Lyman, Samuel Nunez, Zayna Osborne, 
Sabrina Wilson, & Emily Chavez 
California State University, Fresno 


SIOP UN Committee: Mathian Osicki, Julie Olson-Buchanan, Lise Saari, Nabila Sheikh, John Scott, Lori 
Foster, Deborah Rupp, Mary O’Neill Berry, Walter Reichman, Dan Maday, Aimee Lace 


Are you interested in using I-O psychology to make a difference in the world? Do you find SIOP’s work 
with the United Nations intriguing and are looking for a way to get involved? If so, the SIOP UN team in- 
vites you to consider working on a new initiative to identify how the science and practice of I-O psychol- 
ogy can inform work on the 17 United Nations Sustainable Development Goals (SDGs). The 17 United 
Nations SDGs are a “universal call to action to end poverty, protect the planet and ensure that all people 
enjoy peace and prosperity” (United Nations Development Programme, n.d.) and includes such goals as 
eradicating poverty and hunger and ensuring health/well-being and quality education, among others. 


As an NGO with special consultative status to the United Nations, SIOP is called on to respond to this ral- 
lying call through our expertise in I-O psychology. However, our field does not traditionally categorize 
our research and practice in terms of the areas identified in the SDGs and, as such, the SIOP UN Team 
has worked on translating how our field might align with and support the SDGs. 


To aid in this effort, we are launching a new initiative. Specifically, we are interested in the creation of 
annotated bibliographies for each of the 17 SDGs. Such bibliographies will enable SIOP to be better pre- 
pared to weigh in on position papers (for which we often have only a few days to respond) and aid in the 
identification of l-O psychology expertise and best practices as it relates to a particular SDG. In addition, 
we believe it will facilitate the growth of our field in the direction of humanitarian work psychology by 
identifying gaps in the literature and stimulating additional research in l-O psychology that serves or 
supports the United Nations mandate. 


In the remainder of the article, we describe the process and preliminary findings for the first annotated 
bibliography we developed. It focuses on SDG #2, which is to “end hunger, achieve food security, and im- 
proved nutrition, and promote sustainable agriculture” (United Nations Development Programme, n.d.). 


Process Used for Pilot Annotated Bibliography 
Form Workgroup 


There are a number of existing workgroups (e.g., research labs, work colleagues) that would work well 
for this initiative, or you can form a new one for this purpose. We formed a group with seven under- 
graduates and the first author at California State University, Fresno. The undergraduates included Psy- 
chology and Business majors, all of whom had taken an Introduction to I-O Psychology course and had 
an interest in the field. However, the possibilities for forming groups are wide-open and can be a great 
way to team build within a local I-O group, work colleagues in different disciplines, or graduate students. 
Select SDG 


We selected the 2"! SDG because hunger is a critical concern in our region. Other groups might select an 
SDG that resonates with them or is most closely linked with their areas of practice or research. To select 
an SDG, please see a list of the 17 SDGs, which are available on the SIOP United Nations 


(http://www.siop.org/Prosocial/UN.aspx) page, and inform Julie Olson-Buchanan (julieo@csu- 
fresno.edu) of your plans to ensure that groups do not duplicate efforts. 


Brainstorm 


At first blush, it is hard to see the link between our research and practice and such goals as eradicating 
hunger. This is where brainstorming in the group is key. Some prompts that helped us identify potential 
literature included “why would or should organizations/employees care about this issue?” and “how 
might something I-O psychology does well relate to a solution for this societal issue?” 


Search for Literature and Complete Initial Annotated Bibliography Entries 


After brainstorming some ideas, we divided up the areas and individuals searched the literature (includ- 
ing journals and conference papers) in traditional I-O psychology areas as well as related fields, and ex- 
amined write-ups of relevant projects in organizations. It is important to take a broad approach here, 
within psychology and related fields, particularly given that there is (generally) limited research availa- 
ble. To keep the annotations consistent, we used a template that included the following components: 
(a) Description and summary of the study’s or project’s purpose and approach (method), (b) summary of 
main conclusions, (c) source’s link to the SDG, and (d) Future research suggestions. 


Revise and Brainstorm Structure of the Annotation 


By posting the annotated bibliographies on a shared drive, we were able to revise and modify the anno- 
tated bibliographies. A key part of the process included meeting again to discuss the findings and brain- 
storm additional directions to search. 


General Findings 


Regular, secure access to nutritious food is a key component to improving current and future conditions 
for our society, particularly with respect to the development of a global workforce. As noted by the Food 
and Agriculture Organization of the United Nations (2015) “Hunger and undernutrition mean less-pro- 
ductive individuals, who are more prone to disease and thus often unable to earn more and improve 
their livelihoods. This, in turn, hinders progress in alleviating extreme poverty and fighting hunger—par- 
ticularly as labor is the principal asset held by the poor” (p. 27). As such, our workgroup identified two 
general areas of l-O research and practice that can serve to inform the United Nations Mandate relating 
to SDG 2. This includes the literature relating to how food security and nutrition is a critical factor for the 
internal functioning and well-being of organizations and the CSR literature and related publications on 
private organizations’ efforts to address hunger issues. 


Food insecurity and nutrition directly affect worker and organizational well-being. Consider, for exam- 
ple, the high number of food insecure workers in a developed country, such as the United States. Ac- 
cording to a 2017 USDA report, 12.3% of U.S. households (or 1 in 8) qualified as food insecure. A 2009 
report found that 68% percent of households with food-insecure children had at least one adult working 
full-time, and 78% had at least one adult working full or part time (United States Department of Agricul- 
ture Economic Research Service, 2009). Fifty-four percent of households that utilize Feeding America 
food banks have at least one member who has worked in the past year, and 20% had members em- 
ployed throughout the year (Babic et al., 2015). As such, there appears to be a significant number of 
workers suffering from food insecurity. Moreover, a healthy diet is more expensive than an unhealthy 


one, at a difference of $1.15 to $1.94 per day (cf. Rao, Afshin, Singh, & Mozaffarian, 2013, for a review). 
Research indicates that in households that experience food insecurity, adults tend to sacrifice quality 
and quantity of food, saving food for children (Coates, et al., 2006) in part because existing food pro- 
grams for children do not appear to be sufficient (Pickett, Michaelson & Davison, 2015). Although the 
percentage of workers in developing and developed countries varies around the world (cf. Food and Ag- 
riculture Organization of the United Nations, 2015), it is clear that hunger is a significant problem for a 
sizable number of adults who are in the workforce, as well as for those who are not in the workforce. 


Consequences of food insecurity in the workplace include outcomes associated with sleeplessness, 
worry, guilt, alienation, and psychological strain (Coates et al., 2016). Working hours are a particularly 
critical time for eating and nutrition (Castellari & Berning, 2016). Yet, some of the most food insecure 
adults have limited access to healthy, affordable food while at work, underscoring the need for healthy, 
inexpensive (or free) food at or near the workplace for a significant portion of lower wage earners. In- 
deed, although research on adults in the work context is limited, healthy eating is related to several pos- 
itive outcomes including affect (White, Horvath & Conner, 2013) and performance (Edwards, Mauch & 
Winkelman, 2011). 


Organizations may engage in corporate social responsibility (CSR) efforts that are designed to address 
hunger or nutrition in their local or global community, such as manufacturers donating overproduction 
to food banks, raising funds for hunger relief, or workers volunteering at food pantries as part of an out- 
reach effort. One particularly compelling example is ConAgra’s five prong strategic initiative to improve 
children’s food security in the United States (Knowlton, Phillip, & Knowlton, 2012). Research has identi- 
fied several positive outcomes of CSR efforts for participating organizations, such as affective commit- 
ment (Mueller, Hattrup, Spiees, & Lin-Hi, 2012), prosocial behavior (Thornton & Rupp, 2016), employee 
recruitment (Catano & Hines, 2016), and positive consumer perceptions (Du, Bhattacharya & Sanker, 
2010). Yet, it is important to consider the context for CSR initiatives, as there is some evidence that con- 
sumer perceptions of CSR in developing countries may not share the same outcomes (Arli & Lasmono, 
2010), and CSR initiatives that are imposed upon developing countries may lead to complicating issues 
within an area of need (Khan, Westwood, & Boje, 2010). 


Organizations can also contribute to humanitarian efforts and societal well-being by sharing their exper- 
tise. For example, a number of studies in related fields have examined how business expertise in logis- 
tics (Kovacs & Spens, 2007) can improve access to food for those affected by disaster (Van Wassenhove, 
2006). The common theme of such research is that the viability of such private sector-humanitarian 
partnerships center on making a compelling case for the consequences of CSR, an area in which I-O psy- 
chology has considerable expertise (e.g., Knowlton & Phillips, 2012; Thornton & Rupp, 2016). 


Future Research Areas 


We identified a number of important areas for future research. In particular, there is little research on 
food insecurity and its effects on job performance and health of workers in developed countries, and 
virtually no research on these variables in developing countries. Interventions in the workplace, such as 
access to inexpensive, healthy food should be examined, particularly with respect to key outcome varia- 
bles such as safety, health, and performance. Relatedly, the relation between food insecurity and em- 
ployment-readiness has not been examined, an issue that might relate to a cycle of poverty and has im- 
plications for available labor pools. There is a substantial amount of research on the antecedents and 
consequences of CSR. Another area for future research in this area could include how the findings of CSR 


research in I-O psychology can be used to facilitate the creation and expansion of private sector-human- 
itarian partnerships. 


Conclusion 


We found our work on the annotated bibliography to be very informative and will hopefully serve as a 
model for the additional work needed to cover the remaining SDGs. It helped us see I-O psychology 
through a different lens and made us better appreciate how I-O research and practice can be translated 
to address and inform actions needed to accomplish the SDGs. The annotated bibliographies will be 
stored ona shared drive for the United Nations team and will be critical resources for informing SIOP 
position papers as well as SIOP’s weighing in on policy making, and special projects that serve the United 
Nations mandate. If you are interested in creating an annotated bibliography on another SDG for the 
UN, please contact Julie Olson-Buchanan at julieo@csufresno.edu. We look forward to expanding our 
partnerships within SIOP to help make a difference in the world through I-O psychology. 
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Call for Nominations 
Nominate the Next Professional Practice Series Editor 


SIOP is now soliciting nominations for the position of editor of the Professional Practice Series. The 
new series editor will be selected by the Publications Board and approved by the Executive 

Board. The selected series editor will begin working with Incumbent Editor Nancy Tippins, beginning 
December 2018, and assumes duty beginning April 1, 2019. 


The series editor must be a SIOP Fellow or Member. Any SIOP Fellow or Member can nominate indi- 
viduals for the editorship. Self-nominations are also welcome. 


Professional Practice Series Description 


The series is designed to provide industrial-organizational psychologists, organizational scientists and 
practitioners, human resource professionals, managers, executives, and those interested in organiza- 
tional behavior and performance with volumes that are insightful, current, informative, and relevant 
to best organizational practice. The series volumes are guided by five tenets designed to enhance fu- 
ture organizational practice: 


1. Focus on practice, but grounded in science. 

2. Translate organizational science into practice by generating guidelines, principles, and lessons 
learned that can shape and guide practice. 

3. Showcase the application of industrial-organizational psychology to solve problems. 

4. Document and demonstrate best industrial and organizational practices. 

5. Stimulate research needed to guide future organizational practice. 


The series volumes seek to inform those interested in practice with guidance, insights, and advice on 
how to apply the concepts, findings, methods, and tools derived from research in industrial-organiza- 
tional psychology to solve organizational and human resource related problems. 


Each volume is focused on one important theme or element of I-O practice and is usually composed 
of 10—15 separately authored chapters related to the overall theme of the volume. The chapter au- 
thors are solicited and selected by the volume editor(s) in consultation with the series editor and the 
series editorial board. The final selection of authors and the editorial quality of the chapters is the re- 
sponsibility of the volume editor(s) who may seek consultation with the series editor as they see fit. 
The intent is to publish approximately one new volume each year. 


Position Description 


The series editor is appointed to a 5-year term and is responsible for managing the production of se- 
ries volumes, including identifying and soliciting volume editors and ideas, and assisting volume edi- 
tors with identifying chapter authors and ensuring chapter quality, as needed. The series editor is as- 
sisted by a series editorial board, which is assembled by the series editor in consultation with the 
SIOP Publication Board. 


Requirements for Series Editor 


It is highly desirable that series editors be experienced practitioners. Practitioners would include I-O 
psychologists in academic or applied research organization positions for whom applied practice is a 
substantial component of their position. That is, university faculty who devote virtually all of their 
time to teaching, research, and service would typically not be recruited as a series editor, unless their 
academic appointment was recent and they had substantial practice experience in a previous posi- 
tion. Beyond being expert in particular areas of practice, the series editor should also have a broad 
perspective of I-O psychology, including both its scientific base and the full spectrum of applications. 


Information for Nominees 


On average the series editor devotes approximately half a day per week to his/her editorial responsi- 
bilities, although the actual time spent in any given week is highly variable. 


The most time-intensive aspects of the series editor position are the identification and recruitment of the indi- 
vidual volume editors and the identification and recruitment of the appropriate chapter authors. Potential vol- 
ume editors can be solicited and recruited in many ways (e.g. issuing a call for ideas and proposals for a vol- 
ume), and it is the series editor’s responsibility to use all available means. The editorial quality of a specific vol- 
ume is the primary responsibility of the volume editor, with advice and consultation from the series editor. 


Other demands include helping to advertise and market the series (e.g. suggesting advertising copy) 
and meeting with editorial board members and volume editors at SIOP, or at other conferences. 


Nomination and Application Information 


If you are interested in serving as series editor, or if you know someone who might, submit your nom- 
ination via email by August 24, 2018 to Mark Poteet (mlpoteet @verizon.net). Nominations need only 
be a brief letter announcing the intention to nominate. Only those individuals nominated by this date 
will be able to submit application materials. 


All nominated individuals should submit their application package by September 28, 2018 to Mark 
Poteet (mlpoteet@verizon.net). Each application package should include electronic versions of a cur- 
rent CV, a statement that describes the nominee’s vision for the Professional Practice Series, and 
three letters of recommendation from SIOP Fellows or Members. 


SIOP Publications Board 

Mark Poteet (Professional Practice Series Editor Selection Committee Chair) 
Richard Klimoski 

Steve Kozlowski 

Ron Landis 

Neal Schmitt 

Nancy Tippins 

Mo Wang 


Call for Nominations 
Nominate the Next Organizational Frontiers Series Editor 


SIOP is now soliciting nominations for the position of editor of the Organizational Frontiers Series. 
The editor will be selected by the Publications Board, and approved by the SIOP Executive Board. The 
selected series editor will begin working with Incumbent Editor Rich Klimoski, beginning December 
2018, and will assume duty beginning April 1, 2019. 


The series editor must be a SIOP Fellow or Member. Any SIOP Fellow or Member can nominate indi- 
viduals for the editorship. Self-nominations are also welcome. 


Organizational Frontiers Series Description 


Launched in 1983 to make scientific contributions to the field, this series publishes books on cutting- 
edge theory and research derived from practice in organizational and industrial psychology, and re- 
lated disciplines. The goal of the series is to inform and stimulate research for SIOP members (stu- 
dents, practitioners, and researchers) and people in related disciplines including other subdisciplines 
of psychology, organizational behavior, human resource management, and labor and industrial rela- 
tions. The series is designed to provide industrial-organizational psychologists, organizational scien- 
tists and practitioners, human resource professionals, managers, executives, and those interested in 
organizational behavior and performance with volumes that are insightful, current, informative, and 
relevant in the conduct of research and practice in the organizational sciences. 


Each volume is focused on one important theme or element of I-O research and is usually composed 
of 10-15 separately authored chapters related to the overall theme of the volume. The chapter au- 
thors are solicited and selected by the volume editor(s) in consultation with the series editor and the 
series editorial board. The final selection of authors and the editorial quality of the chapters is the re- 
sponsibility of the volume editor(s) who may seek consultation with the series editor as they see fit. 
The intent is to publish approximately one new volume each year. 


Position Description 


The series editor is appointed to a 5-year term and is responsible for managing the production of Series vol- 
umes, including identifying and soliciting volume editors and ideas, and assisting volume editors with identify- 
ing chapter authors and ensuring chapter quality, as needed. The series editor is assisted by a series editorial 
board, which is assembled by the series editor in consultation with the SIOP Publication Board. 


Requirements for Series Editor 


It is highly desirable that series editors be experienced researchers who work in academic or applied 
research positions for whom the conduct and application of research in the organizational sciences is 
a substantial component of their position. The series editor should also have a broad perspective of l- 
O psychology, including both its scientific base and the full spectrum of applications. 


Information for Nominees 


On average the series editor devotes approximately half a day per week to his/her editorial responsi- 
bilities, although the actual time spent in any given week is highly variable. 


The most time-intensive aspects of the series editor position are the identification and recruitment of 
the individual volume editors and the identification and recruitment of the appropriate chapter au- 
thors. Potential volume editors can be solicited and recruited in many ways (e.g. issuing a call for 
ideas and proposals for a volume), and it is the series editor’s responsibility to use all available 
means. The editorial quality of a specific volume is the primary responsibility of the volume editor, 
with advice and consultation from the series editor. 


Other demands include helping to advertise and market the Series (e.g. suggesting advertising copy) 
and meeting with editorial board members and volume editors at SIOP, or at other conferences. 


Nomination and Application Information 


If you are interested in serving as series editor, or if you know someone who might, submit your nom- 
ination via email by August 24, 2018 to Neal Schmitt (schmitt@msu.edu). Nominations need only be 
a brief letter announcing the intention to apply. Only those individuals nominated by this date will be 
able to submit application materials. 


All nominated individuals should submit their application package by September 28, 2018 to Neal 
Schmitt (schmitt@msu.edu). Each application package should include electronic versions of a current 
CV, a statement that describes the nominee’s vision for the Organizational Frontiers Series, and three 
letters of recommendation from SIOP Fellows or Members. 


SIOP Publications Board 

Neal Schmitt (Series Editor Selection Committee Chair) 
Richard Klimoski 

Steve Kozlowski 

Ron Landis 

Mark Poteet 

Nancy Tippins 

Mo Wang 


CONFERENCES AND MEETINGS 


Please submit additional entries to Marianna Horn at mhorn@aptmetrics.com. 


2018 


April 12-16 


April 13-17 


April 19-21 


April 22-25 


May 6-9 


May 24-27 


July 28-August 2 


August 9-12 


August 10-14 


October 1-5 


Oct 19-20 


Oct 28-Nov 3 


2019 


May 8-11 


Annual Convention, National Council on Measurement in Education. San New 
York, NY. Contact: NCME, http://www.ncme.org/ncme/NCME 


Annual Convention, American Educational Research Association. New York, NY. 
Contact: AERA, www.aera.net. 


Annual Conference of the Society for Industrial and Organizational Psychology. 
Chicago, IL. Contact: SIOP, www.siop.org. (CE credit offered.) 


HRPS Global Conference. Scottsdale, AZ. Contact: HRPS, www.hrps.org. 


Annual Conference of the Association for Talent Development. San Diego, CA. 


Contact: ATD, http://www.atdconference.org/ 


Annual Convention of the Association for Psychological Science. San Francisco, 
CA. Contact: APS, www.psychologicalscience.org. (CE credit offered.) 


Joint Statistical Meetings. Vancouver, BC, Canada. Contact: American Statistical 
Association, www.amstat.org (CE credit offered.) 


Annual Convention of the American Psychological Association. San Francisco, 
CA. Contact: APA, www.apa.org (CE credit offered.) 


Annual Meeting of the Academy of Management (AoM). Chicago, IL. Contact: 


AoM, http://aom.org/. 


Annual Meeting of the Human Factors and Ergonomics Society. 
Philadelphia, PA. Contact: The Human Factors and Ergonomics Society, 
www.hfes.org. (CE credit offered.) 


14 Annual Leading Edge Consortium. Baltimore, MD. Contact: Society for 
Industrial and Organizational Psychology, www.siop.org/lec/2018. (CE credit 
offered.) 


Annual Conference of the American Evaluation Association. Cleveland, OH. 
Contact: AEA, www.eval.org. 


Congress of the European Association of Work and Organizational Psychology. 


Turin, Italy. Contact: EAWOP, hittp://eawop2019.org/ 


March 20-23 


April 4-6 


April 4-8 


April 5-9 


August 8-11 


August 9-13 


Oct 28-Nov 2 


Annual Conference of the Southeastern Psychological Association. 
Jacksonville, FL. Contact: SEPA, www.sepaonline.com. (CE credit offered.) 


Annual Conference of the Society for Industrial and Organizational Psychology. 
Fort Washington, MD (Washington, DC area). Contact: SIOP, www.siop.org. 
(CE credit offered.) 


Annual Convention, National Council on Measurement in Education. Toronto, 
ON, Canada. Contact: NCME, http://www.ncme.org/ncme/NCME 


Annual Convention, American Educational Research Association. Toronto, ON, 
Canada. Contact: AERA, www.aera.net. 


Annual Convention of the American Psychological Association. San Chicago, IL. 
Contact: APA, www.apa.org (CE credit offered.) 


Annual Meeting of the Academy of Management (AoM). Boston, MA. Contact: 


AoM, http://aom.org/. 


Annual Conference of the Human Factors and Ergonomics Society. 
Seattle, WA. Contact: The Human Factors and Ergonomics Society, 
www.hfes.org. (CE credit offered.) 


IOTAs 


David L. Tomczak 
George Washington University 


Honors and Awards 


Alan Colquitt published Next Generation Performance Management: The Triumph of Science Over Myth 
and Superstition. 


Morrie Mullins received the Bishop Fenwick Teacher of the Year Award at Xavier University. 


Psychology and Work, by Donald M. Truxillo, Talya N. Bauer, and Berrin Erdogan, is a winner of the 
2018 Textbook & Academic Authors Association’s Most Promising New Textbook Award. 


Dave Winsborough published Fusion: The Psychology of Teams. 
Transitions, New Affiliations, Appointments 


Charles Calderwood was named assistant professor of Industrial and Organizational Psychology in the 
Department of Psychology, part of the Virginia Tech College of Science. 


Lisa Nishii was promoted to Vice Provost for Undergraduate Education at Cornell University. 


Good luck and congratulations! Keep your colleagues at SIOP up to date. Send items for IOTAs to Tara 


Behrend at behrend@gwu.edu. 


